10 Tips to Supercharge Your Perforce Hosting
More and more teams using Perforce Helix Core are moving to remote working environments every day.
Some teams are going remote by choice, with members distributed across different geographies and work-from-home increasingly offered as a workplace perk in the name of work-life balance.
Other teams have been suddenly forced into remote work environments by the COVID-19 pandemic and ongoing lockdowns that have been instituted worldwide.
Regardless, remote development isn't anything new to the game development industry. But going remote can make it difficult to collaborate on games because of performance issues that can easily crop up when trying to collaborate over long distances on WAN internet connections.
Couple large data volumes with mediocre internet connections that individual team members are often stuck with at home—distributed across different cities, regions, or even continents—and production can easily grind to a crawl.
While some performance issues are unavoidable when working with large files, the Assembla DevOps team recommends a number of tricks and strategies to optimize your Perforce hosting and retain high performance while going remote.
We have developed this list of strategies over 9+ years of experience hosting Perforce for tens of organizations with hundreds of team members distributed around the globe and thousands of gigabytes (terabytes) of data under version control. Whether you host your own Perforce server or are looking for a managed service partner to take over your server management, continue on to read Assembla's top ten tips for maximizing Perforce Helix Core performance.
Server-Side Performance Tips
1. Migrate your Perforce server to the Cloud.
Migrating to cloud-based depots is key to ensuring high performance for remote development.
Perforce cloud installations typically perform so well that Perforce Consultant Tom Tyler has gone so far as to call them "a little bit of black magic in the cloud.” Cloud-based installations benefit from cutting-edge improvements that the major cloud providers are constantly making to their hardware and networking gear. Equipment of the caliber offered by companies like Amazon Web Services, Microsoft Azure, Google Cloud Platform and others would be grossly cost-prohibitive to procure for a single company and refresh year after year. But economies of scale and virtualization can make them available in the cloud for a fraction of the cost.
Additionally, most modern cloud providers offer data transfer speeds between their global data centers that are significantly faster than VPNs and even most private, dedicated network providers. One of the most important strategies to overcoming poor performance is distributing Perforce server resources close to distributed team members using replicas or edge servers located in different data centers around the world. Leveraging the high-speed global networks maintained by modern cloud providers, localized cloud replicas become an even more effective tool to reduce latency and improve the experience of developers and artists alike.
Finally, more and more studio organizations these days are opting for a lean operating model that minimizes overhead expenses in favor of strategic investment in key focus areas. Teams want to spend money on things that actually make a difference to the quality of production—and their bottom line. Many teams forego hiring any dedicated IT resources at all, and add P4 hosting duties to the long list of other responsibilities already assigned to build or game infrastructure engineers.
Besides being a poor allocation of already-stretched resources, remote work environments make it even easier riskier to add on-premises hardware maintenance to your build team's task list. Your source code and art are the most important assets for a software or game development team, aside from personnel. It is crucial that you ensure uptime, availability, and easily-recoverable backups to avoid incredibly costly server outages and lost production time.
2. Provision the right kind of cloud resources and tune your server for p4d.
Some cloud instances enable better performance than others for Perforce, and it’s important that you provision the right set of cloud resources to support your team’s usage patterns while avoiding cost overages, server swamp, or other performance degradations.
Best to use compute-optimized resources
Based on Perforce benchmarking, cloud-based installations of Perforce Helix Core on compute-optimized resources—such as AWS’ c5 class of instances—have been shown to provide the optimal balance between efficient cloud resource utilization and performance.
Additionally, if you are designing your own cloud-based Perforce infrastructure, you should aim to use instances that:
- - Have high I/O (1 Gigabit minimum, but recommended 10 Gigabit or more)
- - Can support solid-state file storage
- - Come with a minimum of 8 cores and 2.4 GHZ processing speed (or 4 cores and 3.2 GHZ)
- - And are supplied with a sufficient amount of memory (You can estimate the amount RAM your team will need using depot size—1.5 kilobytes per file in the depot—and/or number of users—64 MB per user)
Tuning the server for Higher TCP Throughput
There are often adjustments you can make to the operating system of your server to optimize data throughput.
If you are using Perforce 2017.1 or later on a Linux system, you can enable Perforce autotune to improve performance. For more on Perforce autotune, see here.
If you are hosting on a Linux-based system with Windows-based clients, you can also manually add the following lines to /etc/sysctl.conf or a custom conf file in /etc/sysctl.d/ (if this path is present and referenced by /etc/sysctl.conf ), and then run 'sudo sysctl -p' to load in the new sysctl settings.
net.core.rmem_default = 2796203
net.core.wmem_default = 2796203
net.core.rmem_max = 22369621
net.core.wmem_max = 22369621
net.ipv4.tcp_rmem = 4096 2796203 22369621
net.ipv4.tcp_wmem = 4096 2796203 22369621
Make sure to keep your server version up to date.
While this may go without saying, it is actually quite important that you keep your Helix Core server up to date. With each new release, Perforce makes optimizations and tweaks under the hood to speed up p4d. The longer you put off updating your p4d version, the more performance benefits you’re missing out on.
At the very least, your team should plan to upgrade your p4d server with every yearly major release (although minor releases often include important bug fixes and updates as well and so should be strongly considered).
3. Set server-side p4 configurables.
Besides operating system-level tuning, there are certain configurables that can be set on your Perforce server to reduce the resources and processing time required to facilitate certain commands.
These configurables include (in order of usual level of performance improvement):
- - Parallel processing for submits and syncs
- - Read-only clients for build servers
- - Lockless peeking
- - MaxResults
- - MaxScanRows
- - MaxLockTime
- - MaxOpenFiles
Of these, MaxLimits settings (i.e. MaxResults, MaxScanRows, etc.) should only be set in extreme cases where the size of team and/or repository require it and usage patterns are well-understood by server administrators. If this are not the case, MaxLimits can easily over-restrict users and contribute to production delays and frustration.
4. Limit client views with Protections or Tight Workspace Views.
Although protections are most often used as a way to set restrictions on certain users or groups of users, they can also improve performance by limiting the number of files involved in syncing and other p4 server operations initiated by groups of restricted users. You must have superuser privileges to set protections. For more information, see Setting protections with p4 protect.
"Tight" workspace views
Similarly, you can manually control the amount of data referenced by your clients by creating a “tight” workspace view. In this case, you add more detail about the files that should be mapped from the depot to the client view, omitting files or streams that do not need to be accessed by that view. Then, any users with access to the client view will only reference the files and metadata contained in the view when executing Perforce commands, reducing the total amount of data that needs to be relayed between client and server and reducing server processing time. See Tight views.
5. Take advantage of Perforce Federated Services to distribute cloud resources.
While it can be complicated to deploy and maintain multiple regions of infrastructure, Perforce Federated Services are instrumental to facilitate high-performance Perforce Helix Core installations for a number of reasons.
First of all, replicas and edge servers deployed in distributed geographic regions help to reduce latency caused by geographic distance and drastically improve server performance for team members that aren't located on the same continent as the Master. Part of the reason this is such an effective strategy in the cloud is that the cross-continental fiber-optic networks of today's major cloud providers are the fastest way for data to cross the globe. Cloud-based multi-site Perforce topologies are the most powerful way to deploying multi-site topologies that have ever been possible.
Additionally, specific types of replicated servers aid in the distribution of workload across the system. Proxies, replicas, and edge servers can each serve certain sets of Perforce server operations, reducing the amount of data required to be transferred back and forth to the master or commit server. This is especially the case for build replicas or build edge servers, which prevent build farms from overloading the master server.
6. If you still have a core group of team members in an office, set up a local on-premises proxy.
Although many teams want to get away from maintaining infrastructure locally, hybrid cloud topologies that include a local p4 proxy or replica server can significantly improve local performance if some team members still work from a single central location.
Client-Side Performance tips
In addition to server-side optimizations, there are a number of best practices you can employ on the client-side to improve performance and streamline cloud-based collaboration.
7. Set client-side configurables.
Much like configurable properties on the server-side, Perforce clients can be tuned to maximize data transfer speeds and reduce the amount of data transferred between server and client to improve performance at the client.
Optimize TCP and Buffer size
Perforce uses the TCP protocol to relay data between different Helix Core servers in your network and client machines. Depending on the quality of your local internet connection, you can sometimes improve performance by tuning TCP and buffer settings on your local machine.
You can either change these settings by adding the following lines to your p4config.txt file, or set the properties from the command line.
P4 Command Line
$ p4 property -a -n filesys.bufsize -v 2M
$ p4 property -a -n net.tcpsize -v 2M
Limit the maximum number of files displayed per changelist
Lowering the default value of "1000" reduces the amount of information P4V has to retrieve from the server for each file in the changelist. Select the "Preferences..." menu item in P4V. Select the "Connection" tab ("Server Data" in P4V 2012.1 and greater). Set the "Maximum number of files displayed per changelist" field to a lower number. Click "OK" to close the Preferences window.
Limit P4V polling frequency
By limiting how often P4V checks the Perforce, you can improve the performance of your local P4V client and also reduce the load on the server.
You can update polling frequency from your P4V preferences menu, or by entering the following in the P4 Command Line client:
$ p4 property -a -n P4V.Performance.ServerRefresh -v 60
Note: This P4V variable can be controlled from the server using the p4 property command described in Storing Property Settings in Perforce.
Note: If set to never check for updates, users need to manually refresh P4V by selecting the "View | Refresh All" menu item. Select the "Preferences..." menu item in P4V. Select the "Connection" tab ("Server Data" in P4V 2012.1 and greater). Set the "Check server for updates every NNN of minutes" field to a higher number or zero (to never check for updates). Click "OK" to close the Preferences window.
Minimize server polling in other clients besides P4V
Configure clients, such as Visual Studio, to “lazy load file state” if possible. While this may reduce the speed with which you receive updates, for most workflow situations you don’t need the updates immediately anyway and this can significantly improve the response time of your client and reduce network congestion.
8. Implement a branching strategy that minimizes impact on the server.
You can also improve the performance of your Perforce connection by limiting the number of files you need to download to your machine. One way to minimize the total dataset size is to employ specific branching strategies or branch types, like task streams and virtual streams, that minimize metadata stored about the stream or only include a specific subset of files.
A task stream is a lightweight, sparse stream that is quicker to use than a regular stream. Task streams are only intended to live for a short time in one individual’s workspace and do not create the same amount of metadata as regular streams, so therefore don’t require the same amount of data transfer or back-and-forth communication with the server. For more on using task streams, see Working with task streams.
Virtual streams work similarly to task streams, with some differences in terms of the workflow options available when working with them compared to task streams. Virtual streams are particularly helpful because they allow you to filter files from a stream that you do not need to sync to your client machine, but still allow you to submit changes directly to the parent stream without having to promote your changes back up the stream. For more details on using virtual streams, see Working with virtual streams.
Branching subsets of files
In addition to creating different types of streams, you can speed up the work you need to do on a set of files by creating a branch with the specific subset of files needed for given feature development or bug fix. In this case, you retain all the workflow capabilities of a standard branch, but minimize the amount of data that needs to sync back and forth between your machine and the central repository by selecting a particular set of files to populate your branch. To do this, simply select the specific folder or set of files you would like to branch, right-click, and choose Branch files.
9. Manually specify the files referenced by your Perforce command.
In addition to branching, you can also manually control the amount of data you need to sync from the central repository to your client machine by limiting the amount of data referenced by certain commands you perform.
For example, you can sync smaller numbers of files at a time by syncing specific files in your depot in separate commands.
p4 sync //depot/projA/...
p4 sync //depot/projB/...
10. Use file locks to avoid lengthy resolve processes.
In many cases, using a branch-merge development model with Perforce streams can accelerate your development workflow. But locking files is sometimes the simplest, most bulletproof way to avoid conflicts for large binary files. File-locking is definitely a less modern workflow-style and won't work for every studio, but it can help save you time and headache by eliminating the need to resolve conflicts between binary files later. For more on using locks to avoid multiple resolves, see Preventing multiple resolves by locking files.
Getting started with hosted Perforce Cloud
Assembla is the exclusive provider of on-demand Perforce Cloud—the GitHub for Perforce Helix Core—and the leading provider of managed, single-tenant Perforce cloud hosting. Whether your team is just getting started with Helix Core version control or already has a Perforce server with hundreds of thousands of files and terabytes of data, Assembla's Perforce Cloud hosting solutions are tuned for high performance out of the box.
To request migration assistance, get a quote, or ask us about anything else related to Perforce hosting, please reach out to our Assembla Customer Success Team at firstname.lastname@example.org, or start a chat with us on https://assembla.com/perforce.