Update the API upload and publish tutorial for GUI v2 projects by siamakn · Pull Request #379 · FAIRmat-NFDI/nomad-docs

siamakn · 2026-09-29T11:51:34Z

Updates the "Upload and share with the API" tutorial for GUI v2 projects. for #376.

"project" replaces "upload"/"dataset" in prose (lowercase, as in 367 update uploading and publishing data with the gui tutorial gui v2 projects #374); code keeps the API's "upload" names, explained in an info box (the API endpoints still say "upload", which is backend relevant. see Update the upload and publication how-to for GUI v2 Projects #365).
The dataset section is removed, as datasets should no longer be the primary method (Preparing the docs for GUI v2 #322) and DOIs are now assigned to the project directly (Update the upload and publication how-to for GUI v2 Projects #365). A TODO marks a possible "assign a DOI via the API" section.
The account link in "Before you begin" was already fixed on develop by repointed the account creation links to correct location in How-to s #326.
Tested end to end on the test deployment (using NORTH, on 2026-09-29); example outputs are from this run. Fixes from testing: new-GUI links, waiting for processing, 3-month embargo (the GUI's smallest option), username in the user search, 60 s timeouts.
Important TODO and the reason:
Workaround: the package's test URL currently rejects uploads over 1 MiB; the page redirects it, and a TODO says to remove that once fixed.
Follow-ups are flagged as TODO comments in the page, as in 367 update uploading and publishing data with the gui tutorial gui v2 projects #374.
Screenshots reduced from four to one (Preparing the docs for GUI v2 #322 asks for as little imagery as possible). Screenshot is by default PNG; changing to SVG is possible but is larger. It is currently PNG rather than the writing guide's preferred jpg. JPG was blurred a bit, and had larger size in first try. but can be compressed further if needed.

github-actions · 2026-09-29T11:53:07Z

PR Preview Action v1.8.1
🚀 View preview at https://docs.nomad-lab.eu/pr-preview/pr-379/
Built to branch `gh-pages` at 2026-10-01 13:58 UTC. Preview will be ready when the GitHub Pages deployment is complete.

JFRudzinski

Nice work @siamakn , thanks for addressing this one. We are in a bit of an awkward transition phase. Many of the comments can be deferred to follow-up work by creating issues (here and in the utility repo)

I think most importantly we need to see about using this backend url, I am not sure how safe this is

JFRudzinski · 2026-09-29T19:17:13Z

 ## Environment setup

-In this tutorial, we will use the [NOMAD test deployment](https://nomad-lab.eu/prod/v1/test/gui/search/entries){:target="_blank" rel="noopener"}. Therefore, in all code examples, we will set `url="test"` when calling the helper functions. Later, you can switch to `url="prod"` or a custom NOMAD API URL if needed.
+In this tutorial, we will use the [NOMAD test deployment](https://nomad-lab.eu/test/){:target="_blank" rel="noopener"}. Therefore, in all code examples, we will set `url="test"` when calling the helper functions. Later, you can switch to `url="prod"` or a custom NOMAD API URL if needed. When you switch to `url="prod"`, also set `nomad_gui`, defined in [Create projects](#create-projects), to `https://nomad-lab.eu/prod/v1/gui/v2`.


it doesn't really make sense to link to a section further down the page. Probably it makes more sense to create a section towards the end called something like "using the production server" and then list all adjustments when you go beyond simply testing

Also, these gui addresses should (eventually) be added to the nomad-utility-workflows package ... it would be great if you could open an issue there

Yes, opened FAIRmat-NFDI/nomad-utility-workflows#55.

After concluding the modifications, I removed the link. Since the settings now sit together in the environment setup, I listed the changes for production below them. Please check and see if moving to a section at the end is a better solution.

JFRudzinski · 2026-09-29T19:19:47Z

 # Upload and publish data using the NOMAD API

-In this tutorial, we interact with the NOMAD API using Python and the [`nomad-utility-workflows`](https://fairmat-nfdi.github.io/nomad-utility-workflows/){:target="_blank" rel="noopener"} package, to programmatically perform the full data upload and publishing workflow. We work with example data files to inspect generated entries, modify metadata, organize entries into datasets, and publish the results on the NOMAD test deployment. By the end of the tutorial, we will have reproduced the core upload and publishing workflows available in the NOMAD GUI.
+<!-- TODO: Align the capitalization of "project"/"entry" with the docs-wide decision (#365 uses "Project"/"Entry", #374 uses "project"/"entry"), and add "project" to the glossary. -->


FYI - added via #381

Thanks, the TODO is removed.

JFRudzinski · 2026-09-29T19:23:18Z

 !pip install python-dotenv
 ```
-<!-- markdownlint-disable MD046 -->
+<!-- markdownlint-enable MD046 -->


I do not like all of these linting adjustments throughout, the linting is there for a reason, if an adjustment is needed it should be raised and made at a more global level if possible

since this is already there partially from before, it does not need to be addressed in this PR, but please open a corresponding issue to keep track

Right, I had them in the old page when I couldnt figure out why some tests fail. I remove them here.

JFRudzinski · 2026-09-29T19:24:26Z

-    Username: siamak.nakhaie@physik.hu-berlin.de
-    Email: siamak.nakhaie@physik.hu-berlin.de
+    Authenticated as: FAIRmat Training
+    Username: test_siamak.nakhaie


I guess you could also remove your name from here, this is not validated anywhere right?

Thanks, I agree. They are now placeholders here and in the user-search output further down.

JFRudzinski · 2026-09-29T19:28:44Z

+```python
+from nomad_utility_workflows.utils import core
+
+core.NOMAD_TEST_URL = 'https://nomad-lab.eu/test/backend/api/v1'


@lauri-codes please check this

Thanks! Quick context: the address itself is the test deployment's official API (its config at https://nomad-lab.eu/test/config.js sets api_base_path to /test/backend). The issue is that url='test' in nomad-utility-workflows still points to the old address, which now rejects uploads over 1 MiB, so the tutorial breaks without this line.

I saw 3 ways, and I chose 1 because I thought we want it to be published by tonight.

Keeping the workaround so the tutorial works with the current package, open issues in nomad-utility-workflows, and update the tutorial once fixed.

Hold this tutorial back until the package is updated.

If possible/fine, lift the 1 MiB constraint in the old test address. I checked with the Prod deployment and it was fine. However I stayed with the workaround because I though the tutorial is meant to be performed on the Test.

@lauri-codes @JFRudzinski , I would leave the decision to you.

Thanks for clarifying, why don't we just update nomad-utility-workflows? Can you open an issue there and list all the changes that should be made? Then I can see if we can make a quick update and release, then you could simplify this tutorial again

JFRudzinski · 2026-09-29T19:29:13Z

+`url='test'` in nomad-utility-workflows 0.3.2 points to https://nomad-lab.eu/prod/v1/test/api/v1, which currently rejects request bodies larger than 1 MiB (nginx "413 Request Entity Too Large").
+Without the workaround below, `upload_files_to_nomad` fails with a JSONDecodeError for miscellaneous_data.zip (1.97 MB) and FHI-aims.zip (1.27 MB); xps_nexus_data.zip (0.23 MB) still works.
+The new test API address https://nomad-lab.eu/test/backend/api/v1 and the production API accept these files.
+Once the limit on the old address is lifted, or the package sets NOMAD_TEST_URL (nomad_utility_workflows/utils/core.py) to the new address, remove the workaround below: the sentence, the snippet, and the explanation after it. -->


If you think a change is needed in nomad-utility-workflows, please open an issue

FAIRmat-NFDI/nomad-utility-workflows#56 (the outdated test URL, which causes the 1 MiB upload problem).

JFRudzinski · 2026-09-29T19:30:06Z

+```
+<!-- markdownlint-enable MD046 -->
+
+The address that the package uses for `url='test'` by default currently rejects files larger than 1 MiB, so the uploads below would fail without this line.


If this workaround is kept: I don't think the user needs to know this, just tell them what to do directly, they don't need to know why

Righ, I removed the explanation. The reason is now only in a comment for maintainers, linked to FAIRmat-NFDI/nomad-utility-workflows#56.

JFRudzinski · 2026-09-29T19:31:36Z


+# Base URL of the new NOMAD GUI on the test deployment,
+# used below to build links to projects and entries
+nomad_gui = 'https://nomad-lab.eu/test'


I would collect all configuration adjustments / variable definitions together and add it to the environmental setup section before starting

Done, both settings are now together at the end of the environment setup.

JFRudzinski · 2026-09-29T19:36:08Z

+
+This code triggers the publication action for the DFT project and prints the server response confirming the operation.
+
+<!-- TODO: Decide whether to add a section on assigning a DOI to the published project via the API (/uploads/{upload_id}/action/assign-doi), mirroring "Assign a DOI to your project" in upload_publish.md -->


I would say yes, but it could be deferred to a follow-up PR if you prefer, just make sure you create an issue to not lose track

Exactly, I would defer it and for now kept a TODO for it in the page. the test deployment used in this tutorial cannot assign DOIs, so readers could not run that step yet. But if we insist to have it in the Tutorial, then we can add an admonition, with a warning, and give example to be performed on the production instance. I wouldn't say so, because users might use it for learning/testing purposes, and we dont want that. I guess majority of the users can easily use GUI and a couple of clicks to do so. Specific/advanced users would better to refer to the workflows utility docs.

JFRudzinski · 2026-09-30T08:21:15Z

+The helper `upload_files_to_nomad` both **creates a new project** and **uploads the given ZIP file** to it in a single API call.
+
+!!! info "Projects and uploads"
+    In the NOMAD GUI, you organize your data in projects. The NOMAD API and `nomad-utility-workflows` still use the term *upload* for a project: the API endpoints retain *upload* in their paths, helper functions such as `upload_files_to_nomad` and `get_upload_by_id` work on projects, and the `upload_id` they return is the project ID.


something I just noticed: As far as I can see the new GUI does not use the terminology "project ID", under archive.metadata there is still upload_id, since the overarching metadata display is not yet there, it is unclear what the end results should be ... it seems to me that we should continue using upload_id though since this is the actual quantity stored, @lauri-codes ?

Ah, ok, I see now that under Project > Settings there is a Project ID listed. Just note users can also go to archive.metadata.upload_id ... so maybe for now we want to say upload_id/project_id refer to the same quantity?

Right. For now I updated the note to say that the upload_id identifies the project, that it can be found as Project ID under Settings > General, and that entries store it as upload_id in their metadata.

In case it helps, the new GUI currently uses both labels: "Project ID" (e.g. project settings, Actions table) and "Upload ID" (e.g. the file delete/rename dialogs). I personally think, the fine adjustments of the terminologies used in the tutorial can be addressed later.

…uts, update TODOs, remove lint comments

siamakn requested a review from ahm531 September 29, 2026 11:52

JFRudzinski mentioned this pull request Sep 29, 2026

Preparing the docs for GUI v2 #322

Open

9 of 13 tasks

siamakn requested a review from JFRudzinski September 29, 2026 15:16

JFRudzinski requested changes Sep 29, 2026

View reviewed changes

JFRudzinski reviewed Sep 30, 2026

View reviewed changes

JFRudzinski added the announcement Create an announcement on Discord for this PR label Sep 30, 2026

This was referenced Sep 30, 2026

nomad_gui_url links to the old NOMAD GUI FAIRmat-NFDI/nomad-utility-workflows#55

Open

url='test' points to an outdated test API address FAIRmat-NFDI/nomad-utility-workflows#56

Open

siamakn requested a review from JFRudzinski September 30, 2026 11:05

JFRudzinski force-pushed the 376-update-uploading-and-publishing-data-with-the-api-tutorial-gui-v2-projects branch from a13de03 to 227b171 Compare October 1, 2026 10:29

JFRudzinski mentioned this pull request Oct 1, 2026

Patches for 376 #388

Merged

siamakn and others added 5 commits October 1, 2026 15:50

Update the API upload and publish tutorial for GUI v2 projects

1584b5e

Address review: collect settings in environment setup, anonymize outp…

1b9fc87

…uts, update TODOs, remove lint comments

add waiting helper function

7978cf2

anonymize author ids

9588628

Format the waiting helper and fix indentation in an example output

794a292

JFRudzinski force-pushed the 376-update-uploading-and-publishing-data-with-the-api-tutorial-gui-v2-projects branch from c2e829d to 794a292 Compare October 1, 2026 13:57

JFRudzinski approved these changes Oct 1, 2026

View reviewed changes

JFRudzinski merged commit e73dc61 into develop Oct 1, 2026
5 checks passed

JFRudzinski deleted the 376-update-uploading-and-publishing-data-with-the-api-tutorial-gui-v2-projects branch October 1, 2026 14:50

JFRudzinski mentioned this pull request Oct 1, 2026

Update uploading and publishing data with the API tutorial GUI v2 Projects #376

Closed


		This code triggers the publication action for the DFT project and prints the server response confirming the operation.

		<!-- TODO: Decide whether to add a section on assigning a DOI to the published project via the API (/uploads/{upload_id}/action/assign-doi), mirroring "Assign a DOI to your project" in upload_publish.md -->

Conversation

siamakn commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Built to branch gh-pages at 2026-10-01 13:58 UTC. Preview will be ready when the GitHub Pages deployment is complete.

Uh oh!

JFRudzinski left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

siamakn Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

siamakn commented Sep 29, 2026 •

edited

Loading

github-actions Bot commented Sep 29, 2026 •

edited

Loading

Built to branch `gh-pages` at 2026-10-01 13:58 UTC.
Preview will be ready when the GitHub Pages deployment is complete.

siamakn Sep 30, 2026 •

edited

Loading