I am writing to share my progress on the “Re-import shell commands” project and to make sure I understand the project scope correctly before I start drafting my proposal.
Over the past week, I have been working on the porting process by submitting MRs for missing commands:
Here is a summary of the steps I took to port head:
Copied source from FreeBSD.
Removed OS-specific headers and code not relevant to RTEMS.
Renamed main() to rtems_shell_main_head(), handled exit() via setjmp, and fixed optind state pollution.
Imported missing helper functions like expand_number().
Manually updated shellconfig.h and the build system files.
Verified by running inside sparc-rtems7-sis and comparing output against the host system.
I noticed that some repeated tasks (like renaming main or injecting headers) could be scripted, but the complex logic fixes require manual work.
I have a few questions about the scope:
Porting Strategy: Is the primary goal to increase POSIX coverage by importing missing commands like tail, grep, or is it better to re-import existing commands like ls, cp to bring them in line with modern FreeBSD?
Shell Features: Does the scope include adding shell features like command chaining (pipes) and auto-completion, or should I stick strictly to porting the command utilities?
The primary goal is to bring us in closer alignment with the upstream code. I think this project could easily be expanded to a Large scope project by including a re-import of the entire shell code to replace what we have in RTEMS. This would be quite a large effort, but the end result would migrate shell from cpukit/libmisc/shell to contrib/cpukit/shell or similar, with cleaner tracking to the FreeBSD upstream sources for the primary shell code.
And then the addition of scripts to help import commands etc and add wrappers would be exntesion work from there.
Hi gedare,
I am interested in taking on the larger scope project to migrate the shell to contrib/. I have reviewed the contrib/ README guidelines and understand the workflow required to maintain upstream tracking.
My plan is to follow the standard contrib lifecycle:
Import pure source code from FreeBSD and commit.
Tag the import commit (to mark the upstream version).
Apply RTEMS-specific changes (fixes, wrappers) as separate commits on top.
I have three questions to help finalize my proposal:
For the new location, do you prefer a flat structure like contrib/shell/ or a mirrored structure like contrib/freebsd/usr.bin/ to match upstream paths?
Should I prioritize migrating the existing commands or focus on importing new/missing commands first?
I currently have MRs open for head and true/false targeting the old directory. Should I push to get these merged now, or close them and incorporate them into the new contrib/ migration plan?
Great. I’m not overly worried about the structure but we could definitely discuss this, perhaps open another topic over in General specifically about the contrib area?
You should prioritize migrating the existing commands so that the functionality that we have in place is preserved. As there have been changes made within RTEMS to some of these commands, there might be some variance with the upstream behavior that you’ll have to resolve. It is hard to say until the work actually begins.
I would prefer not to merge changes to the shell now, and instead focus on migration to contrib with the current functionality intact. So my preference would be to close those MRs for now.
Understood. I will close the current MRs for head and true/false and save that work to be integrated later into the new contrib/ structure.
I will open a new topic in the General category to discuss the contrib directory structure as you suggested. I’ll also update my proposal to focus primarily on the migration of existing commands and resolving the upstream variance, with adding new commands as a secondary goal.
While doing some investigation for variance analysis for my proposal to migrate the shell commands to contrib/, I looked into the git history of ls command. I found out that the current RTEMS implementation that was imported in commit 3899a537 is derived from NetBSD, not FreeBSD.
When diffing the current RTEMS main_ls.c against the current NetBSD upstream, the variance wasn’t big, mostly RTEMS-specific wrappers and the setjmp/longjmp exit handling, but when diffing it against the current FreeBSD ls.c the variance was big requiring a fresh port.
So I guess the available options are either:
re-importing these commands from their original source NetBSD upstream and update them.
do a fresh port from FreeBSD for unification across commands.
What are your thoughts on this and which strategy should I use for the migration?
Good catch, we probably should prefer to re-import from NetBSD. As part of your proposal it would then be good to update the table in the shell/README.md with file locations from NetBSD instead of FreeBSD.
Since Amar hasn’t replied yet, I went ahead and did a full audit of the cpukit/libmisc/shell directory in an excel sheet to identify the origin of each command.
I came down to these notes about the commands origins in the directory:
A mix of NetBSD like ( ls, cp, mv ) and FreeBSD like ( hexdump, dd ). These are the core target of the project, I will do a pure-import to contrib/ updating them with the modern versions from NetBSD/FreeBSD and apply the RTEMS specific modifications.
Commands like cat and echo are currently custom RTEMS implementations (BSD-2-Clause). I will propose replacing these with true upstream versions as a stretch to the core target.
I found that main_df.c is GPL-licensed. I will propose prioritizing its replacement with a BSD-licensed upstream equivalent.
Commands like cpuuse and files like shell_script.c will remain in cpukit/libmisc/shell as they are 1st-party code.
Regarding the directory structure (from my other General topic), I plan to follow the structure Amar outlined in Epic #5294 (contrib/cpukit/shell/netbsd/...) and (contrib/cpukit/shell/freebsd/...)
I am now moving forward with writing the formal GSOC proposal based on these phases. Let me know if you have any considerations and comments about this.
Hi @gedare, I saw the note about potentially using ToyBox for the microshell port Joel made in the discussion of #5312. We recently agreed on a strategy to pure-import the existing NetBSD/FreeBSD utilities into contrib/ for my GSoC proposal to fix the current shell’s drift.
Does the potential of ToyBox change our plan for this summer’s contrib/ migration, or should I continue writing my proposal based on the BSD pure-import strategy?
I would encourage you to do the research on both directions and suggest what might be best, or what questions you might have about each way to go. I doubt anyone has looked deeply into it yet.
I used ls as a case study, diffing the current RTEMS main_ls.c against current NetBSD ls.c, and I investigated ls.c and toys.h for the toybox side.
I got these findings:
Both require wrapping globals for the RTEMS single-address space. BSD uses scattered per-command globals, which RTEMS already successfully handles by using per-command global structs and setjmp/longjmp that replaces the exit() calls. Toybox uses a large toys_context and a per-command TT macro deeply coupled across its multiplexer, libraries, and commands, meaning it still requires the same tedious per-function threading work without any real shortcut.
BSD uses standard POSIX fts for file traversal, which RTEMS already has ported. Toybox uses its own custom dirtree library. Porting Toybox means writing a new RTEMS port for dirtree.
BSD uses standard getopt() that is easily swapped for getopt_r() in RTEMS. Toybox uses custom scripts to parse NEWTOY() macros and auto-generate FLAG() constants at build time. Integrating this means rewriting Toybox codegen pipeline in Python for Waf.
Toybox would require porting its custom libraries and rewriting its build pipeline. I recommend we stick to the BSD pure-imports.
I will include this full trade study in my GSOC proposal.
Hi @gedare, a quick question regarding testing scope for the proposal: I ran the “shell01” test suite on the QEMU simulator and noticed it primarily validates the shell engine (login, joel scripts) rather than asserting the stdout of individual utilities like ls or cp.
For validating the new contrib/ pure-imports, is manual/scripted verification against the upstream BSD output sufficient, or is there a hard expectation for GSOC that I write new, automated rtems-test coverage for every migrated utility?