Please use your reply to this blog post to detail the following:
- Please give a full description of the nature of your first AI/ML game project.
- What was the steepest part of the learning curve for this project? Was it learning how to implement the AI or how to use the Pygame/Pyglet library? Please elaborate and explain your answer.
- What went “right” with your project? As in, what worked seamlessly? What went “wrong” with your project? As in, what were your biggest hurdles or where did you have the most trouble debugging or getting your project to run?
- Describe the AI/ML algorithm your game implements. Did you work through a tutorial you found online? Did you start from scratch because you were motivated by a particular game or algorithm and you wanted to implement it using Pygame/Pyglet?
- If you had to teach this class next year, what project would you recommend to students in the Advanced Topics class to give them a broad and comprehensive overview of some fundamental AI algorithms to implement in a game?
- Discuss how your game implements reinforcement learning and how, if you chose to pursue a Deep-Q implementation of reinforcement learning, your game manages game states to learn how to make decisions.
- Include your Github repo URL so your classmates can look at your code.
Take the time to look through the project posts of your classmates. If you saw any project or project descriptions that pique your interest, please reply or respond to their post with feedback. Constructive criticism is allowed, but please keep your comments civil.
Please give a full description of the nature of your first AI/ML game project.
I built a PPO model that used Pymunk physics simulation to build a ragdoll and train it to run and play soccer against another ragdoll.
What was the steepest part of the learning curve for this project? Was it learning how to implement the AI or how to use the Pygame/Pyglet library? Please elaborate and explain your answer.
I would say the hardest part was the Pygame/Pymunk. Setting up the ragdoll was really annoying and when things weren’t working, I was really focused on the RL setup but didn’t think to look at the ragdoll. The problem ended up being purely physics which was annoying. The RL was also difficult to work with because things fail so silently so it’s really hard to debug, but the hardest part ended up being getting the physics of the ragdoll right in Pygame.
What went “right” with your project? As in, what worked seamlessly? What went “wrong” with your project? As in, what were your biggest hurdles or where did you have the most trouble debugging or getting your project to run?
The RL environments were pretty good, I feel like I was able to get those up and running with RlLib without too much hassle. Obviously I had that big problem with the ragdoll which set me back a lot, and I ended up not having enough time to get the soccer working like I wanted it to. I spent so long trying to get them to figure out how to hit the ball the right way but it was just debugging the ragdoll that was important. Once that was done, I just didn’t have enough time to run training for the fully floppy ragdolls, but it was getting promising results at the end. With some more time I could’ve gotten something cooler but I was honestly pretty sick of the debugging at the end because it takes so much time just to see if one small change was effective.
Describe the AI/ML algorithm your game implements. Did you work through a tutorial you found online? Did you start from scratch because you were motivated by a particular game or algorithm and you wanted to implement it using Pygame/Pyglet?
The ragdoll was all from scratch. The ML algorithm was a PPO that took in the ragdoll’s body positions as states and then controlled each joint in order to move the body parts. The ragdoll, when running, was rewarded for moving forward. When playing soccer, it was rewarded for moving towards the ball, hitting the ball, the ball moving towards the goal they were scoring on, and a few various penalties. I started from scratch because I couldn’t really find any ragdoll projects in Pygame, but I had originally wanted to just find one on GitHub so I could focus on the RL. I was glad I made it myself so I could have full control. I didn’t really have any games I was that excited about so this seemed like a cool opportunity which is why I picked it.
If you had to teach this class next year, what project would you recommend to students in the Advanced Topics class to give them a broad and comprehensive overview of some fundamental AI algorithms to implement in a game?
A simple board game or something like that is good for a Q-Learning or more tabular algorithm, and then a really simple motion game would be good for an actor-critic style algorithm. Something like the Dino Game but even simpler could be a good overview, just training a model to jump at regular intervals.
Discuss how your game implements reinforcement learning and how, if you chose to pursue a Deep-Q implementation of reinforcement learning, your game manages game states to learn how to make decisions.
I used a PPO model, which stands for Proximal Policy Optimization. Q learning is a value-based algorithm, basically meaning that it inputs a state and for each possible move, predicts what the reward for that move will be. For continuous tasks like a ragdoll, that is hard to implement because there are essentially infinite combinations of joint velocities so you can’t really split it up into rewards for each move. Instead, proximal policy organization is policy-based, meaning it learns to directly output an action instead of using a set list of actions. It still predicts what the actions reward will be, but it goes straight from state to action instead of deciding all of the possible actions. That makes it much better for continuous tasks. State management was relatively simple, just the position, velocity, angular velocity, and rotation of each body joint. That allowed the ragdoll to be “aware” of itself (and in the soccer game, of the other ragdoll). In the soccer game I also added the ball position and velocity so that the ragdoll knew where to go and what to move towards.
Include your Github repo URL so your classmates can look at your code.
https://github.com/valenmoore/Project02_PygAIme.git
Please give a full description of the nature of your first AI/ML game project.
I made a Lunar Lander style game which uses a PPO reinforcement learning model to learn how to land the lander from a randomized start position and randomized terrain.
What was the steepest part of the learning curve for this project? Was it learning how to implement the AI or how to use the Pygame/Pyglet library? Please elaborate and explain your answer.
Implementing the AI was pretty easy, but some things to make it work better/faster were a bit tough. Figuring out how to correctly normalize from a vectorized environment in order to watch the AI play was pretty hard but ended up about doubling the training speed, so it was well worth it.
What went “right” with your project? As in, what worked seamlessly? What went “wrong” with your project? As in, what were your biggest hurdles or where did you have the most trouble debugging or getting your project to run?
I had the most annoying bug where I had the negative direction of vertical velocity set as up instead of down, which was completely messing up all my rewards structures. For a good few days the AI would just hover at the top of the screen without descending and I couldn’t for the life of me figure out why. Once I found the bug, everything was pretty seamless from there. Things like terrain and VFX took a little bit of thinking about but for the most part were implemented smoothly. Maybe the only big remaining sticking point is that only one action can be done per frame, so you can’t roll the lander and fire the engine at the same time, but it’s only really an issue for manual play, as the AI can just switch between rolling and thrusting every frame and get the same result as doing both at the same time.
Describe the AI/ML algorithm your game implements. Did you work through a tutorial you found online? Did you start from scratch because you were motivated by a particular game or algorithm and you wanted to implement it using Pygame/Pyglet?
My game implements PPO, which is a reinforcement learning algorithm. Every frame, I assign a “reward” to the AI which is intended to tell it if its doing something good or bad. The AI then keeps training until the total reward is as low as possible. I used online tutorials to help with the exact implementation and really it ended up being pretty easy to work into the game.
If you had to teach this class next year, what project would you recommend to students in the Advanced Topics class to give them a broad and comprehensive overview of some fundamental AI algorithms to implement in a game?
I actually think this game is a really good one as its simple enough to where you can understand it pretty easily but also complex enough to where you have to think about your reward structure more than just “losing is bad and winning is good.” Other games, like Tetris or something PvP like connect 4 are just too complex for a single person to really wrap their head around what the AI should be incentivized to do.
Discuss how your game implements reinforcement learning and how, if you chose to pursue a Deep-Q implementation of reinforcement learning, your game manages game states to learn how to make decisions.
The model is rewarded for landing successfully, being lined up laterally with the landing pad, and maintaining a smooth vertical speed. It is punished for crashing, going out of bounds, fuel consumption, and excessive speed or roll angle. It uses PPO to “do” the reinforcement learning and create the model which I then use to watch the AI play.
Include your Github repo URL so your classmates can look at your code.
https://github.com/Freedomplaza/Proj02_PygAIme
I built a reinforcement learning model that utilizes Q-learning to learn how to play Connect 4.
I would say the hardest part of learning in this project was learning how to use keras. A lot of the explanations from flint actually used deprecated code and while sometimes it would work, other times it would just do nothing. I tried to rely less on flint in this project so even though I have a better understanding of what each part of my code does, figuring out how to do some things was tricky. I didn’t really struggle with using pygame because I have experience with it in the past. I knew pretty much what to do with that, which I think helped neutralize the difficulty of this project.
One thing that went well with my project is that I could tell that my models were leaning, which is good. Using a central script to run everything and controlling the project through a GUI helped everything run properly. One of the most annoying parts of the project was just how long it took to run a training session. It really limited my ability to get sufficient training in, especially when they weren’t saved correctly so they weren’t loading throughout the majority of the project.
My AI was made from scratch. I didn’t really look up any tutorials online so i don’t know if there is one I could have followed to make the process easier. There were also a lot of specific elements that I wanted to incorporate that probably wouldn’t have meshed easily with a tutorial, such as the 3 different training modes. It uses q learning, as well as epsilon degradation to try to find the best moves possible while using some risk taking to explore other potential options.
I would tell them to talk to flint. Honestly, Valen’s project would be cool. They could learn about physics, GUIs, and see the results of their training in real time. If they found like a basic tutorial I think this project could help a lot.
Discuss how your game implements reinforcement learning and how, if you chose to pursue a Deep-Q implementation of reinforcement learning, your game manages game states to learn how to make decisions.
My model gets more reward for the longer it stays alive, plus a big reward if it wins and a big negative award if it loses. I made it so it would get a reward for the longer it stays alive so it feels incentivized to actively block the opponent and make more strategic moves because before, it was getting a tiny negative award for how long it stayed alive and because it wasn’t winning, it was just minimizing playing time. Managing games states with connect 4 is pretty clear cut because the model has to take turns. You just reward the model after each time it takes a turn. It can see what the board looks like now, as well as the previous board (right before the opponent made their move).
https://github.com/nataliemcwhorter/Project02_PygAIme.git
1.) Please give a full description of the nature of your first AI/ML game project.
For my project, I built a reinforcement learning google snake game that learned how to actually play the snake game while also trying to get better and better at it.
2.) What was the steepest part of the learning curve for this project? Was it learning how to implement the AI or how to use the Pygame/Pyglet library? Please elaborate and explain your answer.
the steepest part of the learning curve for this project was definitely figuring out my rewards system and my epsilon (how much exploration the model does rather than thought of moves). Because my game was relatively simple at the beginning and gets exponentially harder the longer the body of the snake gets, the model was able to learn how to play very efficiently, it just began to flatline once exploration meant running into itself. In order to fix this, I had to figure out a way for the snake to look six steps ahead of itself, rather than one which came with a lot of implications and sorts because I started to reach a point where I was overcomplicating a “simple” game.
What went “right” with your project? As in, what worked seamlessly? What went “wrong” with your project? As in, what were your biggest hurdles or where did you have the most trouble debugging or getting your project to run?
The thing that went “right” with my project which I am so thankful for is how my model saves, loads, or creates a new one. I can move a saved model to my desktop and have it keep the same features, even if I change the code of the project which was really cool. However, the thing that went wrong in my project was trying to implement a method where the snake learns to not get itself trapped. It was extremely hard because the snake would maybe have to look ten steps ahead, and also it was often counteracted by my other rewards system the model trained on.
Describe the AI/ML algorithm your game implements. Did you work through a tutorial you found online? Did you start from scratch because you were motivated by a particular game or algorithm and you wanted to implement it using Pygame/Pyglet?
The Ai/ML algorithm my project implements is Q-learning, and I did so from scratch because I thought it was the best way for an AI to learn in a simple 2d game where it had discrete choices that it could easily store in a Q-table.
If you had to teach this class next year, what project would you recommend to students in the Advanced Topics class to give them a broad and comprehensive overview of some fundamental AI algorithms to implement in a game?
If I had to teach this class next year, I would definitely not recommend students to do the snake game because I think the fundamental AI algorithms to implement in the are easy to learn from, but they could learn from it within a week, and not a month. I got stuck at a point where I didn’t know how much more I could do to my project to make it better, even though it only averaged a score of ~30.
Discuss how your game implements reinforcement learning and how, if you chose to pursue a Deep-Q implementation of reinforcement learning, your game manages game states to learn how to make decisions.
My game implements reinforcement learning by first finding out its own state(direction, length, food, etc….), then choices an action and gets a reward or gets punished based on if the action was “good” or “bad”. Immediately after the machine chooses an action, the game updates a Q-table which begins to increase in size, allowing the model to combine immediate reward and possible future rewards so it gets smarter and learns more about the moves it should make.
Include your Github repo URL so your classmates can look at your code.
https://github.com/pieter-sauer/Snake-AI-Agent.git
Please give a full description of the nature of your first AI/ML game project.
My project was pretty much a rip-off Geometry Dash that used a q-learning reinforcement learning model to learn how to complete custom levels and survive in infinite levels.
What was the steepest part of the learning curve for this project? Was it learning how to implement the AI or how to use the Pygame/Pyglet library? Please elaborate and explain your answer.
I had a similar experience to Leonardo. Actually implementing the model was easy, because Flint wrote it for me. The hard part was having to learn how the model works and edit it. The template that Flint gave me wasn’t very good and neither did it fit very well with the other file. Fine tuning the code over and over took a long time and a lot of trial runs before I was kinda able to figure out some numbers that worked. The other super hard part was having to scroll through hundreds of lines of code to look for something when I had very little clue what I was looking for.
What went “right” with your project? As in, what worked seamlessly? What went “wrong” with your project? As in, what were your biggest hurdles or where did you have the most trouble debugging or getting your project to run?
The AI worked well when it was running a custom map. It learned the pattern of the map after thousands of runs, and thus ran perfectly in the end. The parts that didn’t go as well was the infinite world section and when I tried to use a model trained on one map on a different one. My models didn’t use Q-Deep, so they had pretty much no adaptive capabilities. All it could really do was learn certain patterns in a map and go off of that. So in infinite mode it would get past the first few obstacles but once it encountered a state it had never been in before, which is quick common, it would break. Writing the GUI was probably the hardest and most annoying part of the project because it required me to write so much code—mainly current state blocks and copying and pasting the basic CUI class flint wrote for main menu for all the other menus while changing things like variablse—and had so many variable names that I had to remember. Because of all this code that was so long, the fact I’m relatively new to python, and the blocks in the main loop looked super similar to each other, it was really hard to debug and fix minor mistakes because it was super hard to find out what i needed, and then fix it and then correct all the problems that change caused in the other files.
Describe the AI/ML algorithm your game implements. Did you work through a tutorial you found online? Did you start from scratch because you were motivated by a particular game or algorithm and you wanted to implement it using Pygame/Pyglet?
So what I did was make a Q-learning model. It relies on a reward based system, where certain things will give or take points away. For example, in a platformer game, moving forwards would give 1 point, but dying would take 100. The model starts out by making completely random moves, then entering them into a table. The table has states(basically what the games looks like to the model at as specific point) and actions(what that model can do at a particular point), and as the model goes through more states and tries out different actions, it fills in each cell with the amount of points in made from doing that action at that state. Over time, the model will act less and less randomly, preferring paths that it knows will return a higher reward.
I started from scratch, planning to use just FlintAI as a resource. The game only had 2 options—jump or don’t jump—so I figured it would be pretty easy to make. Flint sorta walked me through how the code worked, and then wrote a template, which I fine tuned with Claude.
If you had to teach this class next year, what project would you recommend to students in the Advanced Topics class to give them a broad and comprehensive overview of some fundamental AI algorithms to implement in a game?
I would say they should choose a project where most parts of it are simple, with one complex part of it. For example, if you limit the options the AI has(jump or don’t jump), you have the flexibility to test out other areas of the game. Looking back, it would have been cooler and I would have learned more if I used both a q-table and deep-Q on my project, then compared their effectiveness by having one game that has a constant map and one game where the map/objects in the game vary each time it is run. In short, a game like geometry dash or flappy bird, because the simplicity of the game gives them more time to dedicate effort to the AI and trying out different models.
Discuss how your game implements reinforcement learning and how, if you chose to pursue a Deep-Q implementation of reinforcement learning, your game manages game states to learn how to make decisions.
My table uses a q-table. which stores information on how certain states and actions made during those states increase the models total amount of points. Certain results, “rewards”, give different amounts of points. Since this was a Geometry style game, I had 4 rewards set up: winning = +1000, dying = -100, progression through 1 state(basically 1 frame) = +0.1, being in the air for 1 state = -0.05. This made the model prioritize winning over everything, then distance, then being on the ground. After each episode, the information that is learned is added to the q-table, and the next episode will be slightly less random with its choice. I also gave the model a pretty high discount factor of .95 to make sure it would prioritize total points(which includes potential future points) over just maximizing points in the next state.
Include your Github repo URL so your classmates can look at your code.
https://github.com/LanceHackman/Project02_PygAIme
Please give a full description of the nature of your first AI/ML game project.
My first game project was a Tetris AI model that uses reinforcement learning implemented in python with the pygame library. The goal was to create a Tetris-playing agent that could learn to improve its performance through repeated gameplay without having to program explicit strategies.
This project includes a playable Tetris game built with pygame that handles all user interaction and an AI agent that demonstrates improvement through reinforcement.
What was the steepest part of the learning curve for this project? Was it learning how to implement the AI or how to use the Pygame/Pyglet library? Please elaborate and explain your answer.
The hardest part of this project was designing and working with the AI, not programming the game itself. Pygame was relatively easy to learn but implementing a working learning algorithm required a deep understanding of machine learning. the biggest challenges were designing and effective reward function and translating the Tetris board state into something a computer can learn from.
What went “right” with your project? As in, what worked seamlessly? What went “wrong” with your project? As in, what were your biggest hurdles or where did you have the most trouble debugging or getting your project to run?
The AI often failed to improve – the score would stay at 0. This was frustrating because it wasn’t clear whether the problem was in the code or in the logic. I had to learn how to debug both technically and conceptually and think about what information the AI could’ve been missing. Often times, it seemed that no matter what I changed and tried differently it just wouldn’t work. Successful aspects included: the best recorded score rose from 50 to 9800 points, it just wasn’t consistent at all; the q-table expanded steadily; all code worked well, manual Tetris and ai Tetris both didn’t have bugs. Challenges included: the reward system being too simple; the state representation was too complex, making it hard to find patterns; long training times; limited time.
Describe the AI/ML algorithm your game implements. Did you work through a tutorial you found online? Did you start from scratch because you were motivated by a particular game or algorithm and you wanted to implement it using Pygame/Pyglet?
I implemented the Q-Learning algorithm guided by the articles on the project assignment and Flint’s advice. The algorithm works by updating a q-table which stores the expected value of each possible move given the current state of the board. Over time, the agent will use rewards and penalties to strengthen strategy.
If you had to teach this class next year, what project would you recommend to students in the Advanced Topics class to give them a broad and comprehensive overview of some fundamental AI algorithms to implement in a game?
Something that could be cool would be to somehow build a multi-agent reinforcement learning project where maybe the agents compete against one another in a simplified strategy game or a cooperative game where they must work together. This project would give students a very broad undesrtanding of how AI agents interact, adapt, and develop strategies.
Discuss how your game implements reinforcement learning and how, if you chose to pursue a Deep-Q implementation of reinforcement learning, your game manages game states to learn how to make decisions.
The state representation encodes the Tetris board into features such as “lines cleared” and “holes”. An action space consists of all possible piece rotations and column placements. There are reinforcement points for clearing lines, and deductions for creating holes, bumpiness, losing the game, etc. I stored learned state-action values in the Q-table for use in future decisions. This set up would hypothetically allow the AI to gradually discover strategies that resulted in a more efficient gameplay but my project didn’t fully succeed at that.
https://github.com/Emma-Bernstein18/Project02_PygAIme