How to get the same utf-8 encoding as Google for Arabic URLs?

Question

Google: https%3A%2F%2Fwww.aljazeera.net%2Fnews%2Fhealthmedicine%2F2019%2F4%2F29%2F%25D9%2584%25D8%25AD%25D8%25AF%25D9%2588%25D8%25AB-%25D8%25A7%25D9%2584%25D8%25AD%25D9%2585%25D9%2584-%25D8%25A3%25D9%2588-%25D8%25AA%25D8%25AC%25D9%2586%25D8%25A8%25D9%2587-%25D9%2587%25D9%2583%25D8%25B0%25D8%25A7-%25D8%25AA%25D8%25AD%25D8%25AA%25D8%25B3%25D8%25A8%25D9%258A%25D9%2586-%25D8%25A3%25D9%258A%25D8%25A7%25D9%2585-%25D8%25A7%25D9%2584%25D8%25AA%25D8%25A8%25D9%2588%25D9%258A%25D8%25B6

Encoding with utf-8, I get the below: https%3A%2F%2Fwww.aljazeera.net%2Fnews%2Fhealthmedicine%2F2019%2F4%2F29%2F%D9%84%D8%AD%D8%AF%D9%88%D8%AB-%D8%A7%D9%84%D8%AD%D9%85%D9%84-%D8%A3%D9%88-%D8%AA%D8%AC%D9%86%D8%A8%D9%87-%D9%87%D9%83%D8%B0%D8%A7-%D8%AA%D8%AD%D8%AA%D8%B3%D8%A8%D9%8A%D9%86-%D8%A3%D9%8A%D8%A7%D9%85-%D8%A7%D9%84%D8%AA%D8%A8%D9%88%D9%8A%D8%B6

How can I get the same URLs as Google's?

In Python I've used the following method to utf-8 encode the Arabic url:

urllib.parse.quote(url.encode('utf-8'), safe='')

This gives the first encoded url above, which ends with D8%B6. Google's however ends with D8%25B6.

If I copy-paste the Arabic URL from a browser window to another i get the url encoding similar to mine, not the Google one:

Answer 1

The way I understand your question, you have a URL such as (from an Al Jazeera page in this case):

https://www.aljazeera.net/news/healthmedicine/2019/4/29/%D9%84%D8%AD%D8%AF%D9%88%D8%AB-%D8%A7%D9%84%D8%AD%D9%85%D9%84-%D8%A3%D9%88-%D8%AA%D8%AC%D9%86%D8%A8%D9%87-%D9%87%D9%83%D8%B0%D8%A7-%D8%AA%D8%AD%D8%AA%D8%B3%D8%A8%D9%8A%D9%86-%D8%A3%D9%8A%D8%A7%D9%85-%D8%A7%D9%84%D8%AA%D8%A8%D9%88%D9%8A%D8%B6

You then want to construct a Google Search Console URL for this page like:

https://search.google.com/search-console/performance/search-analytics?resource_id=sc-domain%3Aaljazeera.net&hl=ar&breakdown=page&page=!https%3A%2F%2Fwww.aljazeera.net%2Fnews%2Fhealthmedicine%2F2019%2F4%2F29%2F%25D9%2584%25D8%25AD%25D8%25AF%25D9%2588%25D8%25AB-%25D8%25A7%25D9%2584%25D8%25AD%25D9%2585%25D9%2584-%25D8%25A3%25D9%2588-%25D8%25AA%25D8%25AC%25D9%2586%25D8%25A8%25D9%2587-%25D9%2587%25D9%2583%25D8%25B0%25D8%25A7-%25D8%25AA%25D8%25AD%25D8%25AA%25D8%25B3%25D8%25A8%25D9%258A%25D9%2586-%25D8%25A3%25D9%258A%25D8%25A7%25D9%2585-%25D8%25A7%25D9%2584%25D8%25AA%25D8%25A8%25D9%2588%25D9%258A%25D8%25B6

So in short, you have a Google Search Console URL and want to add another URL as a query parameter.

Note that the Al Jazeera URL contains many non-ASCII characters that are properly encoded. In your browser's address bar, the URL will likely be displayed as

aljazeera.net/news/healthmedicine/2019/4/29/لحدوث-الحمل-أو-تجنبه-هكذا-تحتسبين-أيام-التبويض

That's not a valid URL but easier to read. When you copy the URL, you get the escaped one with ASCII characters only. That's the one you start with.

So the steps to create the Search Console URL are:

1. Run the Al Jazeera URL through URL encoding. Most programming language provide such a function. Or there are online service like https://www.urlencoder.org/

2. Append the result to the base Google Search Console:(https://search.google.com/search-console/performance/search-analytics?resource_id=sc-domain%3Aaljazeera.net&hl=ar&breakdown=page&page=!)

That's it.

Note that the Search Console base URL has two peculiarities:

1. The page parameter starts with an exclamation mark, e.g. ...&page=!https%3A...

2. For a different domain, the URL needs to be changed as the domain name appears a second time in the URL.

Python code:

import urllib.parse

url = "https://www.aljazeera.net/news/healthmedicine/2019/4/29/%D9%84%D8%AD%D8%AF%D9%88%D8%AB-%D8%A7%D9%84%D8%AD%D9%85%D9%84-%D8%A3%D9%88-%D8%AA%D8%AC%D9%86%D8%A8%D9%87-%D9%87%D9%83%D8%B0%D8%A7-%D8%AA%D8%AD%D8%AA%D8%B3%D8%A8%D9%8A%D9%86-%D8%A3%D9%8A%D8%A7%D9%85-%D8%A7%D9%84%D8%AA%D8%A8%D9%88%D9%8A%D8%B6"
google_base_url = "https://search.google.com/search-console/performance/search-analytics?resource_id=sc-domain%3Aaljazeera.net&hl=ar&breakdown=page&page=!"

final_url = google_base_url + urllib.parse.quote(url)
print(final_url)

Old answer

URL encoding is a tricky business because of mistakes in the encoding design, pecularities of the web servers and mostly because several different cases are usually mixed up.

Also note that most browsers do not display a correct URL in the address bar, but rather a partially decoded, easier to read URL.

The main cases to distinguish are:

1. Insert data with non-ASCII characters into the path of an URL (e.g.: https://ttt.com/FANCY_CHARACTERS/...)
2. Add data with non-ASCII characters as a query parameter (e.g.> https://ttt.com/res/f?f=FANCY_CHARACTERS)

Your case seems to be a special version of case 2, namely adding a URL as a query parameter to another URL.

So let's assume you have a valid URL from whatever source. It already contains encoded characters.

https://www.aljazeera.net/news/healthmedicine/2019/4/29/%D9%84%D8%AD%D8%AF%D9%88%D8%AB-%D8%A7%D9%84%D8%AD%D9%85%D9%84-%D8%A3%D9%88-%D8%AA%D8%AC%D9%86%D8%A8%D9%87-%D9%87%D9%83%D8%B0%D8%A7-%D8%AA%D8%AD%D8%AA%D8%B3%D8%A8%D9%8A%D9%86-%D8%A3%D9%8A%D8%A7%D9%85-%D8%A7%D9%84%D8%AA%D8%A8%D9%88%D9%8A%D8%B6

If you want to add it to another URL, you just need to run it through URL encoding. You don't need to care about Unicode characters as they are already encoded. The URL contains ASCII characters only:

https%3A%2F%2Fwww.aljazeera.net%2Fnews%2Fhealthmedicine%2F2019%2F4%2F29%2F%25D9%2584%25D8%25AD%25D8%25AF%25D9%2588%25D8%25AB-%25D8%25A7%25D9%2584%25D8%25AD%25D9%2585%25D9%2584-%25D8%25A3%25D9%2588-%25D8%25AA%25D8%25AC%25D9%2586%25D8%25A8%25D9%2587-%25D9%2587%25D9%2583%25D8%25B0%25D8%25A7-%25D8%25AA%25D8%25AD%25D8%25AA%25D8%25B3%25D8%25A8%25D9%258A%25D9%2586-%25D8%25A3%25D9%258A%25D8%25A7%25D9%2585-%25D8%25A7%25D9%2584%25D8%25AA%25D8%25A8%25D9%2588%25D9%258A%25D8%25B6

You can now add this URL to another URL, e.g.:

https://fff.com/ttt/qqq?url=https%3A%2F%2Fwww.aljazeera.net%2Fnews%2Fhealthmedicine%2F2019%2F4%2F29%2F%25D9%2584%25D8%25AD%25D8%25AF%25D9%2588%25D8%25AB-%25D8%25A7%25D9%2584%25D8%25AD%25D9%2585%25D9%2584-%25D8%25A3%25D9%2588-%25D8%25AA%25D8%25AC%25D9%2586%25D8%25A8%25D9%2587-%25D9%2587%25D9%2583%25D8%25B0%25D8%25A7-%25D8%25AA%25D8%25AD%25D8%25AA%25D8%25B3%25D8%25A8%25D9%258A%25D9%2586-%25D8%25A3%25D9%258A%25D8%25A7%25D9%2585-%25D8%25A7%25D9%2584%25D8%25AA%25D8%25A8%25D9%2588%25D9%258A%25D8%25B6

Let me know if that's what you wanted to do...