Sony Patent Suggests “Adaptive Graphics” For Cloud Gaming

For many, cloud gaming (sometimes called gaming on demand or gaming-as-a-service or game streaming) is considered to be the future of video games, enabling players with certain hardware restrictions to play even the most graphically intensive games on less-advanced hardware, such as mobile devices. Considering the increase in demand that cloud gaming has seen in recent years, companies like Xbox Game Studios (Microsoft Corporation) and, more recently, PlayStation Studios (Sony Interactive Entertainment) have released their own versions of the feature to capitalise on the demand.

In January 2014, Sony Interactive Entertainment announced PlayStation Now, the company’s then standalone video game subscription service that offered cloud gaming features for PlayStation, PlayStation 2, PlayStation 3, PlayStation Portable and PlayStation 4 games, which could be played on the PlayStation 4 and PlayStation 5, and even PC. However, the feature did require a minimum of 5 Mbps internet connection to achieve satisfactory performance. While PlayStation Now was received rather auspiciously by players, it was recently shut down with the expansion of PlayStation Plus, another video game subscription service from Sony Interactive Entertainment for PlayStation consoles.

With the incorporation of PlayStation Now into PlayStation Plus, it’s no secret that Sony Interactive Entertainment is constantly trying to innovate cloud gaming with more features to entice players, particularly since cloud gaming is only available to players that have PlayStation Plus on the Premium membership, which costs $17.99 a month (rather expensive compared to its competitors, like the Xbox Game Pass Ultimate). One such innovation that the company is seemingly working on at the moment is “adaptive graphics” for cloud gaming based on the internet connection quality of players.

Earlier today, we came across a recently published patent from Sony Group Corporation under the name SONY INTERACTIVE ENTERTAINMENT INC [JP] titled, “ADAPTIVE GRAPHICS FOR CLOUD GAMING,” which was filed in November 2020 but was published just this week. The patent, which was filed at the European Patent Office, reads, “A method is provided, including the following operations: executing a video game by a cloud game machine, the execution of the video game includes rendering gameplay video; processing the gameplay video by a streaming server for streaming over a network to a client device; monitoring connection quality over the network between the streaming server and the client device; responsive to detecting a change in the connection quality between the streaming server and the client device, then adjusting the rendering of the gameplay video by the cloud game machine.”

One of the criticisms that many players have regarding cloud gaming is its inferior playability on poor internet connections. While cloud gaming can reduce the resolution of the game being streamed to make it playable on poor internet connections, it’s still unable to decrease the graphic quality and fidelity of the game itself to avoid reduced performance. It seems like Sony Interactive Entertainment wants to address this issue by adapting the graphics of the game in accordance with the player’s internet connection quality. “While current cloud gaming systems address varying Internet conditions by changing video encoder settings, such changes only affect existing video/audio that has already been generated by the video game, and does not involve the video game itself,” states the patent.

As an example, the patent suggests that video games being streamed through cloud gaming may “render lower resolution textures that are easier to compress” or “not render unnecessary objects at all.” In turn, this could, theoretically, improve the playability of the game itself as significantly lesser data would be transferred between the cloud game machine and the client device. “Implementations of the present disclosure provide better cooperation between the video game and the video streaming solution in order to obtain a higher quality stream to the user,” clarifies the patent.

Additionally, the patent also discusses how the feature can be implemented to detect any sudden changes in the internet connection quality between the cloud game machine and the client device and adapt the graphics of the game being streamed accordingly. “In some implementations, a method is provided, including the following operations: executing a video game by a cloud game machine, the execution of the video game includes rendering gameplay video; processing the gameplay video by a streaming server for streaming over a network to a client device; monitoring connection quality over the network between the streaming server and the client device; responsive to detecting a change in the connection quality between the streaming server and the client device, then adjusting the rendering of the gameplay video by the cloud game machine,” it reads.

The patent describes many methods to adapt the graphics of the game being streamed in accordance with the internet connection quality of the player, such as “encoding the gameplay video in a compressed video format,” “monitoring one or more of bandwidth or latency,” “monitoring packet loss occurring during the streaming of the gameplay video,” “adjusting a frame rate or a resolution for the gameplay video,” “adjusting a level of detail for the rendering of the gameplay video,” “adjusting an amount of content for inclusion in the gameplay video,” and “reduce (or increase) a bit rate of the streaming of the gameplay video over the network,” Which of these methods are used within the feature is determined by the internet connection quality on the player’s end, as the patent describes, “responsive to detecting a change in the connection quality between the streaming server and the client device.”

Furthermore, one rather intriguing implementation that the patent describes is how much content is shown to the player based on their internet connection quality. “In some implementations, another aspect which can be adjusted by the video game itself, e.g. in response to feedback regarding stream quality or network conditions, is the amount of content in a virtual environment that is rendered or shown in the video frames. One way of accomplishing this is by adjusting the viewing distance in the virtual environment.” Hence, the feature may adapt not only the graphic quality and fidelity of the game being streamed but also its viewing distance. The way that the patent describes this is with “depth,” as in in-game objects are restricted to a certain angular viewing distance within the game based on the player’s internet connection quality. If an in-game object is outside the “depth” of the available internet connection bandwidth, it may not be rendered.

However, this isn’t all, as the adaption is not just restricted to the video quality of the game being streamed but also extends to the audio quality and even the difficulty of the game itself. “In addition to making adjustments that affect video imagery, the video game can also adjust the audio which is being rendered by the video game,” states the patent. “Better results can be achieved by adjusting the audio at the video game itself, rather than relying on downstream encoding adjustments.” For instance, if the game is being streamed at high fidelity audio settings, like 5.1 surround sound, and the internet connection quality decreases to be capable of only stereophonic sound, the feature will adjust the audio quality of the game being streamed accordingly so as to preserve the performance. “Generally speaking, in the above described examples, audio quality is reduced by the video game in response to degraded network conditions such as reduced bandwidth. Likewise, it will be appreciated that in the opposite scenario, when network conditions improve, audio quality can be responsively increased by the video,” it further explains.

Likewise, the feature may also adjust the difficulty of the game itself to preserve the experience since inferior performance means that certain situations within the game, such as combat encounters, will likely also become more formidable. “In some implementations, the video game can adjust the game difficulty based on network connection or stream quality (e.g. latency and bandwidth), with the goal of ensuring that the video game remains playable and enjoyable for the user,” states the patent. The methods detailed for this implementation include features like increasing (or decreasing) the deadlines for certain actions within the game, such as such as attacking or killing adversaries, to make the experience simpler for players with poor internet connection quality. Similarly, the feature may also increase the effect of player assistance features, like aim-assist, to preserve the experience on poor internet connection quality.

Of course, these aren’t the only methods mentioned within the patent, as there are many ways that Sony Interactive Entertainment discusses on how to counter different internet connectivity issues, both on the client device and the cloud gaming machine (which can be read within the patent). It’s also worth noting that this patent was only recently published in China and Europe, but was published in the United States back in May 2021. With the announcement of Google shutting down its own cloud gaming service, Google Stadia, in January 2023, it seems like Sony Interactive Entertainment saw an opportunity to capitalise on the demand with innovative features that remain unexplored (yet) by its competitors, mainly Xbox Game Studios and NVIDIA. Exactly how these features are actually executed, if it all, only time will tell.

What do you think about this? Do tell us your opinions in the comments below!

Similar Reads: Ludia Developing F2P Online RPG Based On “Famous” License