This blog post was co-authored by Reza Jadbabaei, Lead Technical Account Manager at Citrix.

In our previous Citrix TIPs blog, my colleague Brendan Lin wrote a great post on choosing the best upgrade path for your organization. In this blog post, we want to share an interesting case we worked on, with a customer performing an in-place upgrade of their Delivery Controllers from Citrix Virtual Apps and Desktops 7.15 CU5 to 1912 CU1.

This customer, like many, has global operations and delivers business critical applications to users in all geos. We believe the lessons learned from this case will greatly benefit other global deployments and help other customers avoid the same issues. We’ll look at their environment, what decisions they took, what went wrong, and how we helped to fix the issue.

The customer’s environment consisted of:

  • One Citrix Virtual Apps and Desktops 7.15 CU5 site with four zones worldwide (two in APC, one in EMEA, and one in North America, which was the primary zone)
  • Four Delivery Controllers for each geo, for a total of 12. Note that APAC has two zones with two Delivery Controllers each.
  • The highest latency is between the APAC and North America zones and is around 300 – 350 ms. Europe to the other zones is around 200 – 250 ms.
  • The SQL servers configured in an always-on availability group and sit in the North America zone.

The goal? Perform an in-place upgrade from 7.15 CU5 to 1912 LTSR. To start, the customer followed our published documentation for the upgrade process. The procedure is as shown here:

Figure 1: In-place upgrade process

The general process is that the 1912 Delivery Controller software is installed first without installing 1912 Citrix Studio on the target first “half” of the Controllers. When Citrix Studio is subsequently launched on one of the upgraded Controllers, it will prompt to start the upgrade of the database schema, as seen here:

Figure 2: Citrix Studio prompts to upgrade

Two options are presented for the database upgrade:

  • “Start the automatic Site upgrade” if your current user account has the required DB permissions.
  • “Manually upgrade this site” generates the SQL scripts for the DBA to run on the SQL server, as shown on the right-side branch in Figure 1.

Please note, if you chose the Automatic Site upgrade method, you will need to have Security Admin permissions on the Database as well as be a full Citrix admin. This is currently a Citrix Studio limitation in that it processes scripts in a “mixed mode” where no matter the changes being made, SA permissions are required. To get around this, Citrix has provided the option to generate a script and run it manually on the SQL server. In this case, DBO will be enough for Schema Upgrades. See our product documentation for more details.

Because they did not want to affect the primary zone in the U.S., this customer decided to do the database upgrade from one of the upgraded 1912 Controllers in the APAC zone, which has a ~300ms latency to the US datacenter, where the SQL server lives. When attempting this, they ran into the below error:

Exception during site upgrade
Error Id: XDDS:45CB0492
Exception:
    DesktopStudio_PowerShellHistory : GetCapabilitiesScript

    Error Source : Citrix Studio
StackTrace: Citrix.Console.Common.CitrixAggregateException One or more parallel operations failed
    at Citrix.Console.Common.CitrixParallel.InternalForEach[TIn](IEnumerable`1 items, Action`1 oper……..……………

Taking a step back, there were a lot of changes made between 7.15 and 1912, including the introduction of new service instances in 1912 that did not exist in 7.15. During the automatic site upgrade process, Citrix Studio uses its Delegated Admin service to validate if we can perform the schema upgrade on the database. Once the schema upgrade part is complete, Studio then must perform some post-upgrade tasks.

These tasks include re-registering all the service instances of all the DDCs in the site, including those four new service instances. This is no simple task as Studio must coordinate service registration between itself and the SQL server and all the other Delivery Controllers and the SQL server. And among other things, four new rows need to be added in the database for the four new service instances.

The combination of all these tasks happening in parallel and high latency to the SQL server caused a timing error to take place, resulting in a deadlock of the Citrix CCS service and the DA services, with no fall back. The key ingredients here were the new service instances and high latency to the SQL server.

Having discovered this, Citrix released a fix in Citrix Virtual Apps and Desktops 1912 CU3 that more gracefully manages this failure scenario. See CVADHELP-16236 for more information. However, we still recommend that upgrades be conducted from the primary zone, which should always have the lowest latency to the SQL database.

Conclusion

Based on the test results, we recommend doing in-place upgrades following these key points:

  • Always perform the database schema upgrade from a Delivery Controller that is local to the active SQL database. This means avoiding database schema upgrades from a Delivery Controller in a remote zone and sticking to the primary zone where the SQL sits, if possible.
  • If you plan on upgrading using the Automatic Site Upgrade method (Studio GUI-based method), make sure you have the SecurityAdmin role permission granted from your SQL team.
  • If you plan to use the SQL script approach, make sure you have the permissions listed in the following CTX article.
  • If you absolutely must upgrade from a secondary zone with high latency to the SQL server, be sure to take a database and Delivery Controller backup before starting. CDF tracing is recommended on all Delivery Controllers in case of an issue.
  • If you are not a Priority Support customer, and have business critical support needs, we highly recommend Priority Support. With Priority Support, you can get better case prioritization, an assigned Technical Account Manager who will understand your environment and help plan and track your success objectives. Learn more about Citrix Customer Success Services.

Still not sure about performing this type of in-place upgrade for your extremely complex environment with very high uptime SLA? Do not worry. Citrix Consulting Services is always here to help. We will design a specific customized approach for you to upgrade your site including but not limited to providing you with customized help to migrate the configurations between the old and new site as a parallel upgrade. Contact your Citrix sales rep today!