Cracking the Vault

DatabaseError [109]

You can sometimes see this error in the Vault Server vlog. (Note DatabaseError [109] is a generic error. The Exception(Inner) message is more indicative.)

Error: Soap Exception ( mesg-id = xxxxxxxxxxxxxxxxxxxx )
Exception: DatabaseError [109] 
Stacktrace: 
Server stack trace: 


Exception rethrown at [0]: 
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Connectivity.Document.Services.FolderService.AddFile(Int64 folderId, String filename, String comment, DateTime lastWrite, FileAssociationParam[] associations, BOM bom, Stream file, FileClassification fileClass, Boolean hidden)
   at Connectivity.Web.Services.v17.DocumentService.AddFile(Int64 folderId, String fileName, String comment, DateTime lastWrite, FileAssociationParam[] associations, BOM bom, FileClassification fileClassification, Boolean hidden, ByteArray`1 fileContents)

Exception(Inner): Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.
Stacktrace(Inner):    at Connectivity.Core.Database.TransactionContext.OnSqlException(SqlException e)
   at Connectivity.Core.Database.SqlAccess.ExecuteReaderInternal(SqlCommand cmd)
   at Connectivity.Core.Database.SqlAccess.ExecuteReader(CommandType commandType, String commandText, SqlParameter[] commandParameters, Int32 commandTimeout)
   at Connectivity.Core.DataAccess.OwnershipAndAccessControlCheck.ProcessReadOnly()
   at Connectivity.Core.DataAccess.DirectorExtentions.PostBuild(ArrayList builders)
   at Connectivity.Document.DataAccess.Folders.AddFile(Int64 folderId, String name, String comment, Int32 checksum, Int64 fileSize, Int64 createUserId, Nullable`1 createDate, DateTime lastWriteTime, FileAssociationParam[] associations, BOM bom, FileClassification fileClass, Boolean hidden, Boolean unmanaged)
   at Connectivity.Document.BusinessLogic.FolderBL.AddFile(Int64 folderId, String filename, String comment, Int64 createUserId, Nullable`1 createDate, DateTime lastWrite, FileAssociationParam[] associations, BOM bom, Stream file, FileClassification fileClass, Boolean hidden, Boolean unmanagedFile)
   at Connectivity.Document.BusinessLogic.FolderBL.AddFile(Int64 folderId, String filename, String comment, DateTime lastWrite, FileAssociationParam[] associations, BOM bom, Stream file, FileClassification fileClass, Boolean hidden, Boolean unmanagedFile)
   at Connectivity.Document.Services.FolderService.AddFile(Int64 folderId, String filename, String comment, DateTime lastWrite, FileAssociationParam[] associations, BOM bom, Stream file, FileClassification fileClass, Boolean hidden)
   at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Int32 methodPtr, Boolean fExecuteInContext, Object[]& outArgs)
   at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg, Int32 methodPtr, Boolean fExecuteInContext)

It will also normally be accompanied by a Client Side User Interface error dialogue message.

This normally indicates a performance bottleneck issue that needs investigating.

It can be alleviated by increasing the SQL timeouts in the web.config file. Please refer the Advanced Configuration Guide for the version of Vault that you are using for more information.

Limiting Users Connecting to A Vault Server

In some situations it is necessary to limit the numbers of Users connecting to a Vault Server.

 

This can be achieved in a number of ways, such as using different licence servers for each Vault Server. In this blog article, I am discussing Limiting Web Site Usage by throttling the number of Concurrent Connections.

In Internet Information Services (IIS) Manager, on the Default Web Site, in the Advanced Settings, is an expandable node called "Connection Limits". there is a setting "Maximum Concurrent Connections", which in IIS 7.0, has a default setting of "4294967295". Enforcing a non-default setting requires an IISRESET.

Changing this value will limit the number of connections at anyone time to the Vault Server. When the limit is reached, a user will receive a message like this.

"The request failed with HTTP Status 503: Service Unavailable"

Notes 

Each Vault Client application will consume 1 connection. (Example 1 machine with 1x Vault Thick Client and 1x Inventor, runing and logged into Vault, will consume 2 connections)

Logged in but inactive clients release connections, which is then re-acquired following any client-server communication, such as browsing.

When mulitple users are connected to a Vault, the above error message will persist until the total number of connections to the Vault server drops below the defined limit, or the user logs into another Vault Server.

Vault Professional will poll the Vault Server every 5 minutes, see "Vault Licence Release Behaviour"

 

Monitoring Current Connections.

You can use the Windows Performanc Monitor to see the number of users who are connected to the Vault Server using the Web Service\ Current Connections Counter.

The instance selection will depend on if there are other websites running on the same server. For default Vault Server installations, selecting either the "_Total" or "Default Web Site" should be sufficient.

Not Possible to Check in Doc Files in the Vault and CPU of w3wp.exe goes to 100%

A customer reported an issue that they were not able to checking *.doc files into the Vault (while *.docx files were working).
If they tried it the checkin didn´t proceed and was freezing.

Also the w3wp.exe on the Server had a CPU usage of 100% and IIS needs to be restarted.

The cause was a file called OFFFILT.DLL in the folder C:\Program Files\Common Files\Microsoft Shared\Filters

We are not sure where the file comes from but after moving/deleting the file the issue was gone

Useful Performance Counters

These counters are useful if you need to analyse performance bottlenecks in your Vault Environment.

SQL
LogicalDisk(D:)\Avg. Disk Queue Length
Memory\Available KBytes
Memory\Page Faults/sec
MSSQL$AUTODESKVAULT:Buffer Manager\Buffer cache hit ratio
MSSQL$AUTODESKVAULT:Databases(_Total)\Transactions/sec
MSSQL$AUTODESKVAULT:Databases(tempdb)\Transactions/sec
MSSQL$AUTODESKVAULT:General Statistics\User Connections
MSSQL$AUTODESKVAULT:Locks(_Total)\Average Wait Time (ms)
MSSQL$AUTODESKVAULT:Locks(_Total)\Lock Requests/sec
MSSQL$AUTODESKVAULT:Locks(_Total)\Lock Wait Time (ms)
MSSQL$AUTODESKVAULT:Locks\Lock Waits/sec
MSSQL$AUTODESKVAULT:Memory Manager\SQL Cache Memory (KB)
MSSQL$AUTODESKVAULT:Memory Manager\Target Server Memory (KB)
MSSQL$AUTODESKVAULT:Memory Manager\Total Server Memory (KB)
MSSQL$AUTODESKVAULT:SQL Statistics\Batch Requests/sec
MSSQL$AUTODESKVAULT:SQL Statistics\SQL Compilations/sec
PhysicalDisk(_Total)\Disk Reads/sec
PhysicalDisk(_Total)\Disk Writes/sec
PhysicalDisk(0 D:)\Avg. Disk Queue Length
Process(_Total)\% Processor Time
Process(sqlservr)\% Processor Time
Processor(*)\% Processor Time
System\Processor Queue Length

IIS\ADMS
W3SVC_W3WP(#####_DefaultAppPool)\Requests / Sec
W3SVC_W3WP(#####_DefaultAppPool)\Active Requests
W3SVC_W3WP(#####_DefaultAppPool)\Total Threads
W3SVC_W3WP(#####_DefaultAppPool)\Active Threads Count
W3SVC_W3WP(#####_DefaultAppPool)\Maximum Threads Count
W3SVC_W3WP(#####_DefaultAppPool)\Current File Cache Memory Usage
W3SVC_W3WP(#####_DefaultAppPool)\Maximum File Cache Memory Usage
W3SVC_W3WP(#####_DefaultAppPool)\Output Cache Current Memory Usage
W3SVC_W3WP(#####_DefaultAppPool)\Current Files Cached
W3SVC_W3WP(#####_DefaultAppPool)\Current Metadata Cached
W3SVC_W3WP(#####_DefaultAppPool)\Total Metadata Cached

CPU
Processor(_Total)\% Interrupt Time
Processor(_Total)\% Privileged Time
Processor(_Total)\% Processor Time
Processor(_Total)\% User Time
System\Context Switches/sec
System\System Calls/sec

Hard Disk
PhysicalDisk(_Total)\% Disk Time
PhysicalDisk(_Total)\% Disk Read Time
PhysicalDisk(_Total)\% Disk Write Time
PhysicalDisk(_Total)\Current Disk Queue Length
PhysicalDisk(_Total)\Disk Reads/sec
PhysicalDisk(_Total)\Disk Writes/sec
PhysicalDisk(_Total)\Avg. Disk Bytes/Read
PhysicalDisk(_Total)\Avg. Disk Bytes/Write

Memory
Memory\Available Bytes
Memory\Cache Faults/sec
Memory\Demand Zero Faults/sec
Memory\Pages/sec
Memory\Transition Faults/sec
Process(inetinfo)\% Processor Time
Process(w3wp)\% Processor Time
Process(w3wp)\Handle Count
Process(w3wp)\ID Process
Process(w3wp)\Private Bytes
Process(w3wp)\Thread Count
Process(w3wp)\Virtual Bytes
Process(w3wp)\Working Set

Network
Network Interface(Name of Network Interface card)\Bytes Total/sec
Network Interface(Name of Network Interface card)\Current Bandwidth

Performance Issue caused by IIS Logfiles

We received one positive result deleting all current IIS logs on C:\inetpub\logs\LogFiles\W3SVC

After deleting the performance issues were gone.

We are not sure whether it was the fact that they got 15000 log files or many logfiles with a huge size (or both together).

However it is worth deleting old logfiles anyway.

You also could optimize some settings of the IIS Logging:

Performance Issues caused by ESET NOD32 Antivirus

We get more and more reports about customers that do have performance issues, instabilities in the Vault ,frequently errors which is caused by the ESET NOD32 Antivirus.

If you are using this Antivirus and have such kind of issues you need to uninstall the Antivirus on the Server and the Client in order to identify if the issue is caused by ESET Nod32 Antivirus.

Important: in all reported cases it was not enough to disable the Antivirus!

Solution is either to use another Antivirus software or work together with ESET to configure NOD32 that it works propertly together with Vault.

Update 17th October: we got feedback from customers that the problem is solved with the latest version of the ESET NOD32 Antivirus!

Also you have to ensure everything to do with Autodesk products are listed under Exclusions & Excluded applications under Protocal filtering.
Thank you at Vince B. for sharing this information!

 

Improving Grid Population Performance in any Vault Client

If you have a large number of files in a folder and navigating between folders is taking a long time, here's a handy tip.

Select the folder containing large numbers of files. Then right click the column header and select "Customize View".

Click "Custom Filters"

 Click the Basic Tab and enter a wildcard search of *

Click OK.

Custom Filters is then followed by "Any property contains*"

Click "Close" and that folder will have now use the Search "Paging" functionality and a yellow ribbon is then dispalyed at the bottom with a "More" button.

Now, whenever you navigate to that folder, it should not take so long to populate the grid.

 

If you want to avoid setting this up for all folders, you can define a Custom View that can be applied without having to configure it each time.

In the Toolbar at the top of the Vault Interface, expand the pulldown arrow under "Default View"  and click "Define custom views..."

It is a wish to implement this in a future release in a more user friendly way, but for now I wanted to share in case you are struggling with this in Vault 2012.

Knowledge is key!

Autodesk regularly publishes White Papers regarding frequently asked-for topics such as how something works, best practices and optimising performance. There is a wealth of topics covered on the Autodesk Vault White Paper website.

The latest addition to the Vault White Papers website covers tests carried out on Vault speed performance used in conjunction with Riverbed WAN Optimisation technologies. See "Speed collaboration using Autodesk Vault and Riverbed WAN Optimization".

"Geographically dispersed teams using 3D for clear design communication may struggle to collaborate effectively because sharing data using common Microsoft Windows file directories can take too long. Autodesk® Vault helps solve this problem by improving access to the latest designs even across global, wide area networks (WANs). When WANs exhibit high latency, the combination of Autodesk Vault and Riverbed® WAN optimization technology utilizing Steelhead®, a powerful data store and deduplication appliance, provides an effective solution to help maintain fast and efficient design data access. Tests performed by Autodesk illustrate the value of this combination—the addition of WAN optimization may dramatically reduce file transfer by up to 99% without further requirements to upgrade or replace existing software or hardware"

Is it possible split the Vault filestore?

A question that came in regarding a customer who had a very large Filestore.

They wanted to know if it is possible to split the filestore over several disks.

Vault only supports placing the filestore on a single volume.

 

I have previously blogged on the recommended RAID configuration here http://crackingthevault.typepad.com/crackingthevault/2009/07/what-is-the-best-raid-configuration-to-use-for-vault-server.html

Serious Issues like Performance and/or Crashing of the Vault Client

This is a reminder as we constantly receive this kind of requests.

Issue which appears suddenly on all workstations: the Vault Client Application is either crashing (usually after a few Minutes working with the Vault) or you experience serious performance issues.

Root of the issue is that the IT department introduced a proxy Server in the company. Usually a specific proxy script is introduced.

Solution: Make sure you configured the proxy Server as described in the Vault Wiki.

If you need to have a specific proxy script running it is your responsibility to configure it properly that it bypasses all requests about the communication between the Vault Server and the Vault Clients.