Problems I faced when creating the fire workflow

[Leahh02]

I had never written a Dockerfile where I was kind of starting from scratch. I tried to copy the pennant build example but some of the tasks failed before the graph task. So then I skipped those tasks that failed and there were issues with building the container. It would be nice if we could have a Dockerfile that is just the basics of what you need for a python application. Here are some things for the basic Dockerfile:

• What image do you pull? I used almalinux:9 because the pennant build example uses alamalinux and the most recent version of the container we have in beeflow/data/cwl/bee_workflows is version 9.
• What basic things do you need to install? I knew python3 would need to be in there, but are there any other basics to have? My dockerfile probably has some things in there that I don't actually need
• How do you make the python environment?
• How do you copy files into the container? I still haven't fully gotten this to work. If you look at my docker file you can see that it has COPY workdir/fire-workflow/preprocess.py where workdir is in my home directory. The only way I was able to copy files into the container was by specifying the path location from my home directory. At one point I thought I fixed this problem because it was building a container when I changed the path, but it was just using the charliecloud image that I didn't delete. I know the pennant-build workflow copies files into the container but I don't think that actually works.
• I had to make the shell scripts executable

What I did to run the python files was I made a shell script that took in arguments, activated the python environment and then ran the python file with the arguments. Running the shell script was the base command for the task. Is this the best way to do it? I was just following the pennant build example because I don't think we talk about this in the documentation at all.

I had each of my python files use argparse to take in the inputs I wanted to pass to them. Is this the best way to do this?

If I'm using the CWL API and I have a file path as an input, for example: csv_path = Input("csv_path", "string", "./data/data.csv", prefix="--csv_path"), the value for the input has to have more than one forward slash or else it assumes that value is an output from a different step. So "./data/data.csv" is fine but "data/data.csv" is not.

In the script that uses the CWL API I defined common inputs at the top instead of redefining them for each task. Is this the recommended way?

In the optimization steps I pass the parameter grid for gridseach cv to search through as a string input. When I wrote out the value for the parameter grid as an input it looks like this: "'{\"max_depth\":[3,5,7,10,None],\"n_estimators\":[50,100,200]}'". So I know this works, but I don't know if there's a way to write it without all those quotation marks. I think the parser was doing something a little funky.

At one point I was trying to capture array outputs using BEE but I don't think we can. I had mentioned this in the CWL mattermost.

Here's a message I sent to the CWL mattermost.
failing.py is what I want to do:
In the preprocess step the user defines where to save the preprocessed csv as a input Input("csv_path", "string", "./data/data.csv", prefix="--csv_path"), and then that is defined as an output Output("cleaned_data", "File", glob="$(inputs.csv_path)"),. Then in the split_area task I want to give that preprocessed csv as an input Input("preprocessed_csv", "File", value="preprocess/cleaned_data", prefix="--csv_path"),. Only thing is that in the split-area-7121.sh the csv_path argument shows up like: --csv_path $(inputs.csv_path) instead of showing up the way I would like: --csv_path ./data/data.csv
In success.py I have what I've been doing to get around this problem:
just like in failing.py, Input("csv_path", "string", "./data/data.csv", prefix="--csv_path"), and Output("cleaned_data", "File", glob="$(inputs.csv_path)"), are defined in the preprocess step. Then in split_area I give it the user defined csv_path Input("csv_path", "string", "./data/data.csv", prefix="--csv_path"), and I give it the stdout from the preprocessing step Input("preprocess_info", "File", value="preprocess/preprocess_stdout", prefix="--preprocess_info"),. This preprocessing stdout is really just a dummy variable to make sure that split-area doesn't run until preprocess is done running

I know that we can now change the working directory for the task. When I was writing this workflow that wasn't fully a thing so I had the had the paths for where to save outputs as one of the inputs. So for example, right now I have csv_path = Input("csv_path", "string", "./data/data.csv", prefix="--csv_path") for the preprocessing step and the python file saves the csv to that csv path by doing final_df.to_csv(args.csv_path, index=False). But what could happen is I set the data directory as the working directory for the task and just have the python file just do final_df.to_csv("data.csv", index=False). Only problem is the preprocessing task also outputs the encoder to the models directory and I don't want to save the encoder in the data directory.

[Leahh02]

I think one of the roots of my problems as I was trying to make the workflow was that I couldn't find a simple example of a containerized bee python workflow. We have the pennant-build workflow, but that wasn't fully working for me and it's a bit complicated. I've attached a python workflow that is pretty simple (and fun!).

It follows Rusty's suggestion here. Only problem is that it has a dummy input so the say task is dependent on the chmod task.

I like how this example only uses one python package so the dockerfile is pretty basic (It still probably has extra things it doesn't need in it though)

Question: Do we even need to make a python virtual environment if we're running in a container? I'm just doing it because the pennant-build example did it. If we didn't need to then we wouldn't need the chmod task
cowsays.tar.gz

[Leahh02]

bee-cow.tar.gz
I made another example real quick. This one is pretty much the same as the last one, but this one has another task that uses cowsay to print out an ASCII bee saying "Hello from the BEE!"

This one is a little more complicated since it has a second step and the say.py file is a bit longer so it can use the custom art, but I think it's really cute.

The code was edited by chatgpt to handle custom characters that are either in files or given as a literal. I got the ASCII bee from here