UV smoothing in Maya
Software Used:Autodesk Maya
There has been discussion again on Maya highend3d mailing list about the UV distortion caused by polygon smoothing in Maya. I have been working on this problem and devising solutions for a recent production, and as it's hard to explain verbally in a reasonably short email I thought it would be best to sum it up in some tutorial.
WHY DOES MAYA CREATE UGLY UV DISTORTIONS WHEN SMOOTHING POLYGONS?
When you apply a polySmooth in Maya, new vertices will be created, as well as new UV coordinates for these vertices. Both the space coordinates and the UV coordinates of these new vertices will be derived from the old vertices (the original or low res mesh). The problem is whereas new vertices position are obtained through smooth interpolation, the new UV coordinates are obtained through simple linear interpolation. This is the cause of disrependancies between these new vertices space coordinates and UV coordinates and thus, map distortion. If I'm telling you something that you already know, then you can jump directly to the How to make it work part and skip some of the babbling!
To give a visual example of the difference between smooth and linear interpolation, you can see the effect on a smoothed mesh when setting the polySmoothFace continuity parameter to 0. You can see below the difference between linear and smooth interpolation on the same original mesh:
Linearly interpolated shape
Usually you'll model a low resolution version of it and smooth it using Maya's polySmooth with maximum continuity (that is 1). The new vertices positions will be nicely interpolated and you'll get a smooth model, but the UV coordinates will not, they'll just be linearly interpolated you will get texture stretching.
Let's take the example of a polygon head I modeled some time ago, it's a rather light model, nothing fancy in the way it's built : polys are all 4 sided, some vertices are part of 5 edges (which is unavoidable for organic models made of 4 sided polys) and in some occurrence 6 (I'd avoid this actually). You can see the mesh below, and it behaves allright when smoothed as far as volume is concerned:
And that is the UV layout as it has been done (credits to mapper Alban Orlhiac here) on the low resolution mesh:
If we use a grid texture to check UV distortions, we can check how texture will behave on the smoothed object, yuck!
What Maya does
What we would like to see
If we take a close up look at the cheek area you'll notice immediatly the texture stretching artefacts caused by the use of polySmooth:
And Good One
To understand what is happening at UV level, let's check how the smoothed object UV layout compares to the original object UVs:
The polySmooth generated UVs
The UV that would fit the smoothed surface
The UVs generated by polySmooth do not fit the smoothed surface because they're just linearly interpolated (see the straight lines in the left image), and they'd need to be smoothly interpolated the way the mesh is, if we superpose the UV layouts we can see where distortions will happen:
Original UV are in white, bad ones in red, and good ones in green.
HOW TO MAKE IT WORK?
So the problem is that Maya is not smoothing vertices UVs the same way it smoothes vertices space positions (actually it doesn't smooth UV at all). The idea is then to create a new mesh that expresses UV as space coordinates, smooth that mesh, then use the resulting vertices space coordinates to specify the UVs of our final model.
To create a mesh object that represents the original mesh UV coordinates in 3D space, luckily Imre Tuske created a great plugin, called "xyPolyTransferUV" that you can find on Highend3d here.
There is some documentation in the header of the associated MEL script xyPolyTransferUV.mel, by selecting your original (unsmoothed) mesh and calling his script with this command : xyPolyTransferUV(0) you'll create a flat object representing the UV space of the orginal mesh:
After that, just duplicate the original mesh and the flattened object, and smooth both of them using the same continuity and divisions parameters. Set keep tesselation to on, you might want to set keep border to on also or you will have to rework map edges.
Now we have this smooth UV object that represents the UV coordinates we ideally would like see on our smoothed model, because Maya interpolated the position of the new points using it's space smoothing method. Notice how the smoothed flattened UV object differs from the UV layout on the smoothed head:
UVs on the smoothed head
The smoothed flattened UV object
What we have to do is to transfer the UV information back from 3D space to the smoothed head model. Though it is theorically possible to do it with xyPolyTransferUV, there are several issues that keep us from using this plugin to do so. Either it will fail when discontinuous UV hulls are present like in this example, or it will recreate a new high resolution geometry with the correct UV, but then will make it hard to integrate in a full skinned, blended object history like we use for animated characters.
First we'll have to map the flattened UV object so that it's UV coordinates match its space coordinates. It's quite straightforward as the X and Z coordinates of the flattened object correspond to the UV coordinates they represent, thus for a normalised UV layout (set in the 0-1 UV range), a simple planar projection along the Y axis will produce the desired result:
The projection parameters
And the resulting UVs
Then using Maya's menu command Polygons/Transfer and checking only UV sets in the option box, we can transfer back the UVs from the flattened object to the smoothed head:
The Polygons/Transfer options
And the smoothed object with nice UVs
Here we have a way to obtain correctly smoothed UVs, and to apply them back to the smoothed object even when it has a complex history as for animated characters (in that case, the polyTransfer node should come last in the history). Any divisions or continuity parameter can be used for the smoothing as long as both the base mesh and the flattened UV object are smoothed with the same parameters (it can be a good thing to link the parameters of both polySmooth nodes so you can modify one and have the other update automatically). The polyTransfer node can generate messed UVs as smooth resolution is adjusted up or down, so it's a good idea to delete/re-create the polyTransfer node before the final render.