Cracking the Vault

Connected Workgroups - Data Non-Covergence

Referring to TS20200238

"File and File Properties are missing or differ between Publisher and one or more Subscriber Workgroups."

Problem

In a Connected Workgroup environment using SQL Replication, you notice that there are differences in files and file properties between a Publishing and one or more Subscribing Workgroups.

Solution

To avoid this from occurring, Autodesk strongly recommends that you follow the guidance offered in Microsoft Article ID 2591692 “Non-convergence or data loss if a query times out during merge replication in SQL Server 2008 or in SQL Server 2008 R2”. All customers using Connected Workgroup SQL Replication should consider installing this Microsoft update.

 

WARNING. Before any major update in a production replicated environment ensure that you have backed up your publisher and any subscribing databases. Follow the instructions from the “Backup_and_Restore_Autodesk_Vault.pdf”. Refer to the sections “Backing up a Full Replication Publisher”, “Backup Process for the Publisher” and “Backup Process for the Subscriber”

 

RestrictionsOccurred [1387]

This message can appear in the vlogs.

 

Error: Soap Exception ( mesg-id = 635071590603372501 ) 
Exception: RestrictionsOccurred [1387] 
Stacktrace: 
Server stack trace: 
at Connectivity.Product.BusinessLogic.ItemBL.CheckRestrictionsByItemIterationIds(ProductRestrictionCodes[] restrictionCodes, Int64[] itemIterationIds) 
at Connectivity.Product.BusinessLogic.ItemBL.EditItemRevisionInternal(Int64 itemRevId, Boolean bypassWIPRestriction, String comment) 
at System.Runtime.Remoting.Messaging.Message.Dispatch(Object target) 
at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg) 

Exception rethrown at [0]: 
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg) 
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type) 
at Connectivity.Product.Services.ItemService.EditItemRevision(Int64 itemRevId) 
at Connectivity.Web.Services.v18.ItemService.EditItem(Int64 itemRevisionId) 
at SyncInvokeEditItem(Object , Object[] , Object[] ) 
at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs) 
at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc) 
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc) 
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc& rpc) 
at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet) 

The client can see this:-

 

"Operation cannot be performed because the item is locked for edit by user ‘<vaultusername>’."

In this situation, it occured whilst a user was editing an Item, but the Connectivity.VaultPro.exe was killed in Task Manager to simulate a client crash mid-way through editing the Item. The error in the vlog is generic. When the Vault client was restarted, Right clicking on the Item and selecting Edit resulted in the above message in the User Interface and the entry in the vlog.

In this situation you would need to wait 30 minutes before being able to edit the Item. The reason this locktime is enforced is because when an Item is edited, it needs to be temporarily locked to allow the user time to complete their task without the risk of another user overriding their edits. Without this lock, it would be possible that a user might start editing the Item, get distracted and then continue editing it, but when they finish editing it, another user may also have tried editing that Item. At this stage Vault could either overwrite the second users edits or display an error saying that the Item has changed. This would lead to confusion. So to prevent this from happening, if an Item is edited, it is locked until either the user cancels their edit or completes it.

I managed to reproduce this by editing an Item, then killing the process in Task Manager, mid-edit. Similar to a crash, the server would have had no message sent to it that the user has either canceled or completed their task. It assumes therefore that the user is still editing, until the 30 minute lockout expires.

Login Vault Problem: Vault is not compatible with the following applications

We get more and more reports about the (usually false) error message when you try to login a Vault Server from a client in different versions and editions of Vault:
Vault Professional 201X is not compatible with the following applications on MyVaultServer:

Vault Basic Server 18.0.110
Vault Workgroup Server 18.0.110
Vault Collaboration Server 18.0.110
Vault Professional Server 18.0.110

In this situation you made sure that the edition of the Vault Server (for example Vault Collaboration) does match the edition of the Vault Client (for example Vault Collaboration).

In this situation the communication from Vault Client to the Vault Server via Port 80 is partially blocked/disturbed from another application or a faulty DNS entry in the network.

Following causes/solutions are known to this issue:

-different types of Antivirus programs. To make sure Antivirus is not the problem it needs to be fully uninstalled on Server and Client

-Proxyserver running in the environment

-DNS: change the windows hosts file (on client machine) via notepad and add the ip address and host name. You can find more information on this blogpost.

-Windows Firewall blocks the communicatin between Server and Client

-Another application which is using Port 80 blocks the communication

-(Update 20th June): there is another kind of this error where the IIS worker process consume an unusual amount of memory (check the windows event viewer on the Server) and as consequence stopps responding. An IISRESET usually solves the problem

How Much Memory Should I Allocate to the SQL AUTODESKVAULT Instance?

This question occassionally crops up from partners and customers.

The amount of memory needed is often directly linked to the amount of data needed in the cache to achieve high cache hit ratios.

Its not possible to have too much memory for SQL Server, but there must be some left over for the Operating System. So as a general rule of thumb is to have 90% of memory for SQL and leave 10% for the OS to handle general processing.

(The above suggestion assumes that SQL is running on a dedicated server with no other applications such as IIS or another SQL instance is running.)

 

Useful SQL Performance Counters (Source)

• SQL Server: Buffer Manager: Free List Stalls/sec
• SQL Server: Buffer manager: Lazy Writes/Sec
• SQL Server Buffer Manager: Checkpoint Pages/sec
• SQL Server Buffer Manager: Page Life Expectancy
• SQL Server Buffer Manager: Page Lookups/Sec
• SQL Server Buffer Manager: Page Reads/sec
• SQL Server Buffer Manager: Page Writes/ Sec
• SQL Server Buffer Manager: Readahead/sec
• Memory: Free System Page Table Entries

 

Questionable SQL Perforamnce Counters

SQL Server: Buffer Manager: Buffer Cache Hit Ratio

Vault Build Numbers

In the Help>About and Server logs is reported the build number of the version of Vault Server used.

This can help sometimes for you to confirm what version of the software is actually being used if there is some misunderstanding or miscommunication with the servers Administrator.

For example if they are trying to log in Vault 2009 from a Vault 2010 client. Looking at the log files on the server can tell you what version they are actually using.

Vault 2014 (RTM)        18.0.110.x

Vault 2013 (SP1)         17.1.16.0

Vault 2013 (RTM)        17.0.62.0

Vault 2012 (Update 2)  16.2.17.0

Vault 2012 (Update 1)     16.1.58

Vault 2012 (RTM)             16.0.56.X

Vault 2011 (Update 2)     15.2.66.X

Vault 2011 (Update 1)     15.1.58.X

Vault 2011 (RTM)             15.0.58.X

Vault 2010 (Update 2)     14.2.x.x

Vault 2010 (Update 1)     14.1.x.x

Vault 2010 (RTM)         14.0.63.x

Vault 2009 (RTM)         13.0.47.x

Vault 2008 (SP1)         12.1.145.x

Vault 2008 (RTM)         12.0.123.x

 

RTM = Release to Manufacturing, ie, no patches applied

DatabaseError [109]

You can sometimes see this error in the Vault Server vlog. (Note DatabaseError [109] is a generic error. The Exception(Inner) message is more indicative.)

Error: Soap Exception ( mesg-id = xxxxxxxxxxxxxxxxxxxx )
Exception: DatabaseError [109] 
Stacktrace: 
Server stack trace: 


Exception rethrown at [0]: 
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Connectivity.Document.Services.FolderService.AddFile(Int64 folderId, String filename, String comment, DateTime lastWrite, FileAssociationParam[] associations, BOM bom, Stream file, FileClassification fileClass, Boolean hidden)
   at Connectivity.Web.Services.v17.DocumentService.AddFile(Int64 folderId, String fileName, String comment, DateTime lastWrite, FileAssociationParam[] associations, BOM bom, FileClassification fileClassification, Boolean hidden, ByteArray`1 fileContents)

Exception(Inner): Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.
Stacktrace(Inner):    at Connectivity.Core.Database.TransactionContext.OnSqlException(SqlException e)
   at Connectivity.Core.Database.SqlAccess.ExecuteReaderInternal(SqlCommand cmd)
   at Connectivity.Core.Database.SqlAccess.ExecuteReader(CommandType commandType, String commandText, SqlParameter[] commandParameters, Int32 commandTimeout)
   at Connectivity.Core.DataAccess.OwnershipAndAccessControlCheck.ProcessReadOnly()
   at Connectivity.Core.DataAccess.DirectorExtentions.PostBuild(ArrayList builders)
   at Connectivity.Document.DataAccess.Folders.AddFile(Int64 folderId, String name, String comment, Int32 checksum, Int64 fileSize, Int64 createUserId, Nullable`1 createDate, DateTime lastWriteTime, FileAssociationParam[] associations, BOM bom, FileClassification fileClass, Boolean hidden, Boolean unmanaged)
   at Connectivity.Document.BusinessLogic.FolderBL.AddFile(Int64 folderId, String filename, String comment, Int64 createUserId, Nullable`1 createDate, DateTime lastWrite, FileAssociationParam[] associations, BOM bom, Stream file, FileClassification fileClass, Boolean hidden, Boolean unmanagedFile)
   at Connectivity.Document.BusinessLogic.FolderBL.AddFile(Int64 folderId, String filename, String comment, DateTime lastWrite, FileAssociationParam[] associations, BOM bom, Stream file, FileClassification fileClass, Boolean hidden, Boolean unmanagedFile)
   at Connectivity.Document.Services.FolderService.AddFile(Int64 folderId, String filename, String comment, DateTime lastWrite, FileAssociationParam[] associations, BOM bom, Stream file, FileClassification fileClass, Boolean hidden)
   at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Int32 methodPtr, Boolean fExecuteInContext, Object[]& outArgs)
   at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg, Int32 methodPtr, Boolean fExecuteInContext)

It will also normally be accompanied by a Client Side User Interface error dialogue message.

This normally indicates a performance bottleneck issue that needs investigating.

It can be alleviated by increasing the SQL timeouts in the web.config file. Please refer the Advanced Configuration Guide for the version of Vault that you are using for more information.

Illegal pathname or filename during Vault Backup

I have created a Vault environment with a remote SQL (AUTODESKVAULT instance on another machine) which finally went fine and I could work in Vault very well.

However creating a Vault Backup failed with an error message pointing to the SQL Share folder:
The logs gave me following details:
Connectivity.Application.VaultManager.ServiceException: IllegalPath [139] (\\KEVINVMWARE25\SQLShare) ---> Connectivity.Core.Exceptions.CoreException: IllegalPath [139] (\\KEVINVMWARE25\SQLShare)

To solve this I needed to create the AutodeskVault windows user ( with the default password Au1oDSK@26200 ) on the remote SQL machine and give full permissions on the SQL Share folder in the sahring and security tab.
I also needed to make sure that the networkservice has full access on the share as well.

After this the backup finished successfully.

Vault Server 2014 Components installed

Operating System Windows Server 2012 Standard (x64)

• Microsoft SQL Server 2008 (64-bit) (If no version of SQL is pre-installed, SQL 2008 Server (SP3) build 10.0.5500 Express 64-bit, is installed)
• Microsoft SQL Server 2008 Browser
• Microsoft SQL Server 2008 Native Client
• Microsoft SQL Server 2008 Setup Support Files
• Microsoft SQL Server VSS Writer
• Microsoft Visual c++ 2010 x64 redistributable - 10.0.40219
• Microsoft Visual c++ 2010 x86 redistributable - 10.0.40219
• Microsoft WSE 3.0 Runtime
• Autodesk Vault Professional 2014 (Server)
• IIS 8.0 Components

WebServer (IIS)

Web Server

Common HTTP Feature

Default Document

        Directory Browsing

        HTTP Errors

        Static Errors

Health and Diagnostics

HTTP Logging

        Performance

Static Content Compression

Security

Request Filtering

Windows Authentication

        Application Development

.NET Extensibility 4.5

ASP.NET 4.5

ISAPI Extensions

ISAPI Filters

Management Tools

IIS Management Console

IIS 6 Management Compatibility

        IIS 6 Metabase Compatibility

If you need to uninstall the above and reinstall, do so in this order

1. Autodesk Vault Professional 2014 (Server)
2. Microsoft SQL Server 2008 (64-bit)
   1. Database Engine Services
   2. SQL server Replication
   3. SQL Client Connectivity SDK
3. Microsoft SQL Server 2008 Native Client
4. Microsoft Visual c++ 2010 x64 redistributable - 10.0.40219
5. Microsoft Visual c++ 2010 x86 redistributable - 10.0.40219
6. Microsoft WSE 3.0 Runtime
7. Web Server (IIS)
   1. .NET Framework 3.5 Features
   2. .NET Framework 4.5 Features
8. Restart\recycle the machine

Vault 2014 System Requirements

Click this link to view the Vault 2014 System Requirements

 

System Requirements for Autodesk Vault 2014 products

ASP.NET Error Request timed out

In the Windows Event Viewer you can sometimes see these error messages.

Source: ASP.NET 2.0.50727.0

Event ID : 1309

Application Virtual Path: /AutodeskDM/Services

Exception type: HttpException

Exception message: Request timed out

 

This can indicate an ongoing issue that can be immediately resolved by restarting the webservice, but can be resolved by following the advice given in this earlier blog posting...

 

System.OutofMemoryException Messages

ASP.NET Error OutOfMemoryException

Sometime you can see these errors in the Windows Event Viewer Application log.

 

Source: ASP.NET 2.0.50727.0

Event ID : 1309

Application Virtual Path: /AutodeskDM/Services

        Exception type: OutOfMemoryException

        Exception message: Exception of type 'System.OutOfMemoryException' was thrown.

 

In such circumstances, please refer to this former blog posting

System.OutofMemoryException Messages

ASP.NET error The remote host closed the connection. The error code is 0x800704CD.

In the Windows Event Viewer you can sometimes see a number of ASP.NET errors

One such error is

Source: ASP.NET 2.0.50727.0

Event ID : 1309

Application Virtual Path: /AutodeskDM/Services

Exception type: HttpException

Exception message: The remote host closed the connection. The error code is 0x800704CD.

 

 

This happens when a client request information, but cancels the request before it is complete.

 

No action is required from the Administrator

Vault Server (IIS) often down? Restart your IIS regularly

You maybe encounter random problems where Vault is unavailable after some specific actions like checking in a file, copy design etc.

Error messages are for example:
- error 0
- The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.

The consequence is usually that IIS is down and needs to be restarted.

Often the problem is caused by various workflows in the Vault in combination and/or IIS getting out of memory.

We made the experience on different customers that the amount of errors (IIS "crashes") can be reduced if you regularly restart your IIS.

As best practice you could restart the IIS after the daily backup has been finished.

Connectivity Problems solved by Editing the Windows hosts File

We had some cases where we had to "manipulate" the windows hosts file to point to the Vault Server with the IPv4 Address (even though a ping to the Server returned the correct ip Address).

Scenarios where it helped:

-it was not possible to login to the Vault Server from one specific client which got the following error message:
Vault Basic 2013 (17.1.16.0) is not compatible with the data management server found on VaultServername
-connection between 2 sites in a workgroup replication fails with error 26

Changing the hosts file
You can usually find the hosts file here:
C:\Windows\System32\drivers\etc

Open the file via notepad and the following line with then ip4 adress on the left side and the Server name of the Vault Server. For example:
192.168.1.1            VaultServername

...and save the host file via notepad.

To effect the changes you might need to reboot your machine (especially on Windows XP machines).

Attention: please be aware that the IPv4 address of the Server might change if the Server uses DHCP.
So you need to update the hosts file to a later time.

Errors in Vlog about failed Deleting of Files from the Vacuum Cleaner

You may notice that you have lots of entries in the vlog during the night that the Vaccuum Cleaner failed to delete some files in the Filestore:

Error: Vacuum Cleaner: Warning Deleting C:\ProgramData\Autodesk\VaultServer\FileStore\Vaults\Vault\\000\000\000\000\000\189\147854\147854_171149_20.ipt
 Exception: Access to the path '147854_171149_20.ipt' is denied.
Stacktrace: at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
 at System.IO.FileInfo.Delete()
at Connectivity.Core.Operations.VacuumCleaner.TryDeleteFile(FileInfo fi)

Solution:

Make sure that the following users have full access to the complete Filestore:

-Vault Impersonation account (default AutodeskVault windows user)

-Network Service windows account

-Local System account

Limiting Users Connecting to A Vault Server

In some situations it is necessary to limit the numbers of Users connecting to a Vault Server.

 

This can be achieved in a number of ways, such as using different licence servers for each Vault Server. In this blog article, I am discussing Limiting Web Site Usage by throttling the number of Concurrent Connections.

In Internet Information Services (IIS) Manager, on the Default Web Site, in the Advanced Settings, is an expandable node called "Connection Limits". there is a setting "Maximum Concurrent Connections", which in IIS 7.0, has a default setting of "4294967295". Enforcing a non-default setting requires an IISRESET.

Changing this value will limit the number of connections at anyone time to the Vault Server. When the limit is reached, a user will receive a message like this.

"The request failed with HTTP Status 503: Service Unavailable"

Notes 

Each Vault Client application will consume 1 connection. (Example 1 machine with 1x Vault Thick Client and 1x Inventor, runing and logged into Vault, will consume 2 connections)

Logged in but inactive clients release connections, which is then re-acquired following any client-server communication, such as browsing.

When mulitple users are connected to a Vault, the above error message will persist until the total number of connections to the Vault server drops below the defined limit, or the user logs into another Vault Server.

Vault Professional will poll the Vault Server every 5 minutes, see "Vault Licence Release Behaviour"

 

Monitoring Current Connections.

You can use the Windows Performanc Monitor to see the number of users who are connected to the Vault Server using the Web Service\ Current Connections Counter.

The instance selection will depend on if there are other websites running on the same server. For default Vault Server installations, selecting either the "_Total" or "Default Web Site" should be sufficient.