Cargo Space devlog #6 - Steam integration, lobbies, chat

How I got side-tracked into implementing chat, adding Steam lobbies, wrestling logs out of Steam and made a tiny crate for turning crossbeam_channel messages into Bevy events.

This post is part of a series on my game, Cargo Space. It's a coop 2D space game where you build a ship and fly it through space looking for new parts, fighting pirates and the environment.

You can find an overview of the previous entries here

Reliable channels in Matchbox

This story starts with a PR. heartlabs implemented support for adding extra data channels to Matchbox with configurable reliability and ordering guarantees. This got me really enthusiastic about dogfooding the new feature.

If you don't know, Matchbox is a socket library for enabling UDP-like p2p networking in web/wasm (and native) rust applications. This PR extends the scope of the project to also cover reliable channels, i.e. "TCP-like" connections in addition to the existing "UDP-like" ones.

Up until this point, I had done all my networking through bevy_ggrs; I simply handed the matchbox::WebRtcSocket to GGRS and let it do its thing on top of unordered, unreliable messaging. This is sufficient for for general game state, like players moving etc., but for one feature I knew I wanted, it was a poor fit: Chat.

Yak 1: Implementing chat

Since GGRS operates on fixed-size "input" structs (i.e. gamepad input), it doesn't really fit well for chat messages, which occurs infrequently, and can have a quite long payload, and also they don't impact game state in any way, or at least I don't intend them to.

So I had my (only somewhat contrived) use-case.

Extra reliable messages for Matchbox works by adding extra data channel through its brand new builder method (added by garryod)

let (socket, message_loop) = WebRtcSocket::builder("matchbox.example.com")
    .add_unreliable_channel()
    .add_reliable_channel()
    .build();

This creates two channels, one (default) unreliable, and an extra reliable one.

They can then be polled individually through their index corresponding to the order in which they were added:

let new_reliable_messages = let socket.receive_on_channel(1);

GGRS takes ownership of the socket you hand it, but since I didn't want to lose access to polling reliable channels myself, I used an approach suggested to me by zicklag and made a newtype wrapper around the socket, which implements clone:

#[derive(Debug, Resource, Clone)]
pub struct CargoSocket(pub Arc<RwLock<WebRtcSocket>>);

I then implemented the required ggrs::NonBlockingSocket trait for it, simply by calling the inner methods:

impl ggrs::NonBlockingSocket<String> for CargoSocket {
    fn send_to(&mut self, msg: &ggrs::Message, addr: &String) {
        self.0
            .write()
            // if the lock is poisoned, we're already doomed, time to panic
            .expect("failed to lock socket for reading")
            .send_to(msg, addr);
    }

    fn receive_all_messages(&mut self) -> Vec<(String, ggrs::Message)> {
        self.0
            .write()
            // if the lock is poisoned, we're already doomed, time to panic
            .expect("failed to lock socket for receiving")
            .receive_all_messages()
    }
}

This way, the socket wrapper can be cloned. I hand one clone over to ggrs, while the other one I get to keep to send my reliable messages :)

I then created a very simple enum for the game's "protocol":

#[derive(Serialize, Deserialize, Debug, Clone)]
pub enum CargoMessage {
    Chat {
        handle: GgrsHandle,
        message: String,
    },
}

And added simple abstractions for sending and receiving messages on the reliable channel, serializing using bincode:

pub fn send_message(&mut self, peer: impl Into<String>, message: CargoMessage) {
    let bytes = bincode::serialize(&message).expect("failed to serialize message");
    self.inner_mut()
        .send_on_channel(bytes.into(), peer, Self::RELIABLE_CHANNEL);
}

pub fn receive_messages(&mut self) -> Vec<(String, CargoMessage)> {
    self.inner_mut()
        .receive_on_channel(Self::RELIABLE_CHANNEL)
        .into_iter()
        .map(|(id, packet)| {
            let msg = bincode::deserialize(&packet).unwrap();
            (id, msg)
        })
        .collect()
}

pub fn inner_mut(&mut self) -> RwLockWriteGuard<'_, WebRtcSocket> {
    // we don't care about handling lock poisoning
    self.0.write().expect("failed to lock socket for writing")
}

Which means calling code would simply do:

let message = CargoMessage::Chat {
    message: "hello everyone".to_string(),
    handle
};
info!("Sending chat message: {message:?}");
socket.broadcast_message(message); // send to all peers

Adding a UI

Now that I had the networking in place, I needed a UI. While I used kayak_ui last time, and quite like its ideas, I ran into quite a few bugs, for some there exists workarounds, but they are quite ugly, for others like the artifacts on font rendering, I couldn't get it working.

I decided I wanted to use something a bit more mature, and was basically left with the choice between bevy_ui and bevy_egui, both of which I already had in my dependency tree. I don't really like either particularly much, eguis layouts are not flexible enough, and bevy_ui has way too much boilerplate.

However, I decided I'd just pick one and use it temporarily for a throwaway UI, then rewrite it later when the Bevy UI ecosystem is more mature, which will hopefully happen soon, as the Bevy 0.10 release blog post mentioned editor being one of the things to be focused on during the next release cycle.

So since it was a throwaway UI anyway, egui is actually a good choice, since immediate mode UIs are generally much quicker to develop with. And I can live with poor layouts and lack of customization if it's just a throwaway.

So after learning some new API I got a chat box up and running:

Yak 2: Player names

While this is the point where I probably should have turned around and continued work on the actual game, there was one thing the really kind of itched: The players didn't have names. Sure the ggrs handle ids ("Player 0", "Player 1" etc.) serve as passable names, but they aren't nice.

So I added support for adding names through the command line, through --name YourName. However that's not exactly a nice UX, and I was planning/hoping/dreaming to do a Steam release of this game one day, and Steam handily gives you the players nick name, so I kind of thought: "Hey, I'll really quickly setup the Steam integration".

And it was actually quite easy. bevy_steamworks is really easy to setup. And in roughly 5 lines of code, I had access to the Steam user name, so the players get their name with no setup on their part :)

The annoying part was to actually make it work in release builds. I had to copy the shared libraries in the right place on all three platforms, and also make rust binary look for it in the right folder.

Yak 3: Inviting through Steam rich presence

So now that I'd started on the Steam integration, it became too tempting to prematurely do more of the integration work. The next thing I wanted to do was to make it possible for friends to easily join each others games.

One way this is done in Steam, is through the Steam rich presence API.

The idea is that you simply set the command line arguments the other player needs to be able to join your game.

Sounds simple, right? This is what I ended up with:

client
    .friends()
    .set_rich_presence("connect", Some(join_args));

However, it simply didn't work. Nothing happened. I read the docs for set_rich_presence closely and saw that it's supposed to return false on failure. So I added some logging.

if !client
    .friends()
    .set_rich_presence("connect", Some(join_args))
{
    error!("failed to set rich presence: connect");
}

However, that didn't print anything either. So I went digging in the steamworks-rs issues and found a closed issue. Turns out it was a bug that was fixed back in December, but the crate hasn't had a release since then.

So after adding a [patch] entry to my Cargo.toml to make bevy_steamworks use the main branch of of steamworks, I finally got it "working". I could right click a friend, and press "Invite to play":

However, once the player in the other end accepted the invitation, they were met with this:

Turns out Steam does not actually think using command line arguments is secure, and though they don't really explain why. They want you to instead read the command line arguments using their API called GetLaunchCommandLine.

I don't really see how the option they suggest is much more secure, but that's how it is.

Next, it turned out this part of the Steamworks API (GetLaunchCommandLine) was missing from steamworks-rs, so I first had to make a PR to add it

Second, I also want the game to run both with and without Steam integration, and supporting normal command line args at the same time as the ones from Steam was a bit tricky... Especially since Steam's "command line" is just one string, instead of one string per argument, so it kind of maps poorly to tools that operate on regular command line arguments.

I ended up using shlex to turn the string I got from my newly added launch_command_line method back into an array of strings (args), correctly escaping quotes etc. and then feeding that into claps update_from method:

fn update_args(client: Res<Client>, mut args: ResMut<Args>) {
    // get Steam "command line"
    let command_line = client.apps().launch_command_line();

    // the Steam command line does not contain the executable itself (i.e.
    // args[0]), but clap expects it, so we add a fake one
    let full_command_line = format!("cargo_space {}", command_line)

    // clap wants an iterator over the args, not the whole string, so we split
    // it up using shlex (which correctly escapes strings etc.)
    let extra_args = Shlex::new(&full_command_line);

    args.update_from(extra_args);
}

This means any args from Steam basically overrides those from the actual command line. And everything still goes through the same clap struct.

And with that inviting your friends finally worked smoothly.

Yak 3: Lobbies

Up until this point, I'd only supported one type of "lobby", the next rooms from the Matchbox tutorial projects, which will immediately start the game when n players have joined a session. This was fine enough for testing, but when playing with your friends, you probably want to start another behavior.

You don't necessarily know up-front how many players will join
You don't want the game to start until every player has set their status as ready.

And I also wanted this when testing that joining actually worked, to see who joined, who left etc.

So I ended up implementing this type of lobby, and the minimal ui to go with it, using bevy_egui:

The game now simply starts as soon as all players in the lobby have pressed the ready button (start/space).

Yak 4: Steam lobbies

So while inviting now worked correctly, it turned out the "right-click-friend-and-join-game" functionality did not.

I could actually see and click the "Join Game" option, but nothing happened when I did so.

Well nothing is a bit of an exaggeration, but Steam didn't even try to even try to launch the game, the only thing I had to go on was the output from the Steam client itself, which was something like the following:

ExecuteSteamURL: "steam://joinlobby/1234567/false/109775243842407186

I did some searching, and found out join links are not supposed to have false in them. They are supposed to have a lobby id, which put me on the right track; It turns out that in order for "Join Game" to work, you need to use a completely separate system, the Steam matchmaking API.

In order to use it, you have to first create a Steam lobby. Luckily that API was already available through steamworks-rs/bevy_steamworks:

client.matchmaking().create_lobby(
    bevy_steamworks::LobbyType::FriendsOnly,
    max_lobby_players,
    |steam_lobby| {
        info!("created lobby: {steam_lobby:?}");
    },
);

And now, finally something happened when right-clicking a user and clicking "join lobby".

That something was not super exciting, though... It turns out Steam tried to launch my game with the following args:

+connect_lobby <steam_lobby_id>

Which is not really what I had in mind, I'd preferred a normal sane unix command line style, matching what I'm already doing with clap.

Who knows why on earth they would choose + as their prefix for named arguments, but they did. And getting that to play nice with clap was pretty hard. I did a lot of unfruitful searching, and I even tried asking ChatGPT about it, and it invented some new API, then furiously insisted I was simply using the wrong version.

I ended up using subcommands with custom names instead, which is kind of a hack:

#[derive(Parser, Debug, Clone, Resource)]
#[clap(
    name = "cargo_space",
    rename_all = "kebab-case",
    rename_all_env = "screaming-snake"
)]
struct Args {
    //...

    #[clap(subcommand)]
    pub command: Option<Command>,
}

#[derive(Subcommand, Debug, Clone, Deserialize)]
pub enum Command {
    #[command(name = "+connect_lobby")]
    SteamConnectLobby { lobby: usize },
}

I guess the easy way out would've been to just replace + prefixes with -- through string manipulation, but I didn't actually think of it until after I'd made done the hack above.

It was also a bit annoying to have to deal with two sets of lobby ids, I already had one room id which I was using for p2p signalling, but the Steamworks API insisted on generating its own ids. So I ended up setting my id, room_id, as data on the lobby:

client
    .matchmaking()
    .set_lobby_data(steam_lobby, "room_id", &room_id)

Which let me retrieve it after joining a Steam lobby, and plug it into the right place:

if let Some(room_id) = client.matchmaking().lobby_data(lobby_id, "room_id") {
    let room_id = room_id.to_string();
    *requested_lobby = crate::lobby::LobbyType::Private(Some(room_id)).into();
    info!("Set requested lobby: {requested_lobby:?}");
}

And finally lobby joining worked smoothly.

To my surprise, it turned out the matchmaking-lobby-way of joining was now used even when inviting to join as well, taking precedence over the "rich presence" connect key I implemented initially. And the +connect_lobby argument was always used... That meant I could simply delete the path generating and setting that key. In fact it was probably pointless to try and make GetLaunchCommandLine line and command line args compatible in the first place... oh well, a lot of time wasted, but I guess maybe I learned something? 🤷

In any case, I'm really happy that through all of this, I managed to keep the Steam integration completely optional. It's a build time cargo feature. And Steam players can still play with people joining through the wasm build.

Yak 5: Wrestling logs out Steam builds

Another hurdle I ran into was that when the game was launched through Steam and errors were logged, or the game crashed, I had no way to actually see what happened.

I turns out Steam used to put stdout and stderr logs in /tmp, but they stopped doing that, probably because some games spammed logs, filling up the /tmp partition with garbage, and some systems have really tiny /tmp partitions (or no tmp partition), probably causing a bunch of issues.

Regardless, today, stdout and stderr logs are just sent to /dev/null, so in order to actually get your logs you have to redirect them elsewhere. The first thing I thought was that this was a job for a tracing layer, the library Bevy uses for logging. However, it turns out it wasn't possible to change the logging behavior in Bevy without completely disabling the default logging... which is a bit unfortunate, I quite like the default logging and want to keep it if possible.

So I made a draft PR making it possible to inject extra tracing layers. However the API to do so was absolutely hideous. Mainly because tracing changes the type of the logging layer for each new layer added, but also because Bevy has no clean way to take ownership of plugin fields in the plugin's build method. geieredgar has a PR that would solve the ownership issues. But sadly it did not make it in time for Bevy 0.10.

I ended up contributing to getting a more targeted PR working, that simply has a config option to log to a file named after some scheme.

In the end, though, I didn't actually end up using it, as it only catches log messages. However, I also ran into panics, which are not actually tracing log messages and needed to be able debug those as well.

It turns out the way to do it, is to add a "launcher" for your game, that wraps the actual process and redirects its outputs to file.

On Linux, this would be quite easy to do with some simple bash commands, but I needed it to work on Windows as well, so I wrote a simple and stupid rust app called logz_plz. It forwards all its arguments to another command, piping its output to stdout.txt and stderr.txt in the working directory, overwriting any existing files (to prevent wasting disk space).

When I'd done this, I realized I could just as well had done that directly in the game, so I ended up with something roughly like the following:

fn main() {
    let args = Args::get();

    #[cfg(not(target_arch = "wasm32"))]
    if args.launcher {
        // `exit`s after the re-launched child-process finishes
        launcher::launch();
    }

    App::new()
    // ...
}

The launcher method simply relaunches the game with the --launcher argument removed, and all other args intact.

pub fn launch() {
    let stdout = Stdio::from(File::create("stdout.log").unwrap());
    let stderr = Stdio::from(File::create("stderr.log").unwrap());

    // remove --launcher to prevent infinite recursive launching
    let mut args = env::args().filter(|a| a != "--launcher");
    let command = args.next().unwrap();

    let status = dbg!(Command::new(command).args(args))
        .stdout(stdout)
        .stderr(stderr)
        .status()
        .expect("io error while running command");

    // stop execution and forward the inner process' exit code
    std::process::exit(status.code().unwrap());
}

And with this, I could finally get some logs, simply by setting the Steam launch command line to include --launcher, through the Steamworks web ui:

And I could finally see (and fix) my Steam-related panics:

thread 'Compute Task Pool (2)' panicked at 'assertion failed: `(left != right)`
  left: `""`,
 right: `""`', cargo_space\src\steam_integration.rs:143:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'Compute Task Pool (2)' panicked at 'A system has panicked so the executor cannot continue.: RecvError', C:\Users\Johan\.cargo\git\checkouts\bevy-899673624796a25d\7b9124d\crates\bevy_ecs\src\schedule\executor\multi_threaded.rs:194:60
thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', C:\Users\Johan\.cargo\git\checkouts\bevy-899673624796a25d\7b9124d\crates\bevy_tasks\src\task_pool.rs:376:49
thread 'Compute Task Pool (2)' panicked at 'A system has panicked so the executor cannot continue.: RecvError', C:\Users\Johan\.cargo\git\checkouts\bevy-899673624796a25d\7b9124d\crates\bevy_ecs\src\schedule\executor\multi_threaded.rs:194:60
thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', C:\Users\Johan\.cargo\git\checkouts\bevy-899673624796a25d\7b9124d\crates\bevy_tasks\src\task_pool.rs:376:49

Another type of errors I couldn't catch through the Bevy log-to-file feature, was clap args parsing, because I tried parsing args before calling App::run, and even before trying to relaunch the child process, since I needed check for the --launcher arg.

So I wrapped the args parsing, and logged its output to file when it borked:

match Args::try_parse() {
    Ok(args) => args,
    Err(err) => {
        let args: Vec<_> = std::env::args().collect();
        let message = format!("Failed to parse command line args: {err}\n{args:?}");
        if let Err(write_err) = std::fs::write("stderr.log", message) {
            eprintln!("Failed to log to file: {write_err}");
        }
        err.exit();
    }
}

That would leave me with something useful in stderr.log whenever Steam tried to launch my game with its ridiculous args:

Failed to parse command line args: error: unrecognized subcommand

["cargo_space.exe", "+connect_lobby", "109775243842407186"]

Debugging even weirder crashes

So with my fancy new --launcher flag, I thought I'd fixed my Steam logging issues for good... Turned out I was so very wrong.

When I launched my game on Windows my laptop, it still immediately quit, without leaving logs or any trace of why it failed.

So my guess was that, somehow, the launcher was panicking really early.

In the end, I switched back to using my separate logz_plz launcher, which I knew was not panicking. And it did create stdout.log and stderr.log... but there was nothing in them.

I tried running steam-distributed executable from a terminal, but got no output at all, not even a panic, which was really weird.

So I finally tried running the game through gdb, and I saw that it had a weird exit code:

[Inferior 1 (process 3677) exited with code -1073741795]

I tried searching for that, and found a bunch of folks with various kinds of games crashing. As it turns out for some applications it happens because Intel processors had a bug in their microcode, which was fixed in a Windows update... Which would make a lot of sense in my case since the Windows installation on my laptop is borked and no longer updates.

So being a pedantic yak-shaver, I updated my launcher and added a bunch of Steam specific args to it. Even added a log file for the launcher itself, so I could see what I tried to launch, and what exit code it had, and whether it was killed by a signal or not.

I also made it possible to replace the executable itself, as I couldn't find a way with the Steam "Launch options" to replace the executable but keep the args.

When I did so, I found out that Bevy tries to look for assets in the executable's directory, not the current working directory, but you can override it by setting an env var, BEVY_ASSET_ROOT, so I also added support for setting that in my launcher.

Finally, I could set steam launch options like this:

C:\.cargo\bin\steam_dev_launcher.exe -e BEVY_ASSET_ROOT=C:/dev/cargo_space/ --custom-exe C:/dev/cargo_space/target/debug/cargo_space.exe -- %command%

...and I could get Steam to properly launch my debug builds, giving me a launcher.log file with content like this:

16:57:00 [INFO] Parsed launcher args: Args {
    env: [
        (
            "BEVY_ASSET_ROOT",
            "C:/dev/cargo_space/",
        ),
    ],
    custom_exe: Some(
        "C:/dev/cargo_space/target/debug/cargo_space.exe",
    ),
    steam_command: [
        "C:\\Program Files (x86)\\Steam\\steamapps\\common\\Cargo Space\\cargo_space.exe",
        "--launcher",
    ],
}
16:57:00 [INFO] Launching "C:/dev/cargo_space/target/debug/cargo_space.exe" instead of "C:\\Program Files (x86)\\Steam\\steamapps\\common\\Cargo Space\\cargo_space.exe"
16:57:00 [INFO] Launching game: "C:/dev/cargo_space/target/debug/cargo_space.exe" "--launcher"
16:57:03 [INFO] Exited with status 0 (no error)

Or in the case of the weird crash on my laptop, it would give me the strange exit code:

15:57:36 [ERROR] Exited with error status code: -1073741795

The launcher is available here, if you think it sounds useful/interesting.

Yak 6: `bevy_crossbeam_event`

When dealing with the callback style API of steamworks-rs, I found that very often what I most often wanted to do when a callback happened, was to fire a Bevy event with the payload. However, that was slightly tricky, as the callbacks needs to take ownership of anything you pass into them.

I got around this by using crossbeam_channels to send the data back out to a receiver in a Bevy resource that was read in a system that fired the actual event using a Bevy EventWriter.

This quickly got pretty boilerplatey, so I made a generic version of it that adds the event and sets up the channel in one extension method call, and made a new crate, bevy_crossbeam_event.

Usage is like this:

app.add_crossbeam_event::<LobbyJoined>();

And you then get a cloneable resource you can hand over to your callbacks, i.e.:

fn setup(service: Res<ThirdPartyCode>, sender: Res<CrossbeamEventSender<LobbyJoined>>) {
    let sender = sender.clone();
    service.join_lobby(id, move |lobby| {
        sender.send(LobbyJoined(lobby));
    });
}

...and just read the events like normal Bevy events (which they are):

fn handle_lobby_joined(mut lobby_joined_events: EventReader<LobbyJoined>) {
    for lobby in lobby_joined_events.iter() {
        info!("lobby joined: {lobby:?}");
    }
}

Updates

That's it for now. I've done even more yak-shaving and bug-fixing than last time. I'm hoping to get back to some actual game development soon.

For future updates, hopefully gameplay-related, join the discord server or follow me on Mastodon

Comments

Loading comments...