Maya is not known to be helpful when it comes to manually skinweighting a rig. It's default smoothing options aren't very good, and unless you limit how many influences can control every joint as well as keep on top of which joints are locked and which ones aren't, you're most likely going to have issues with Maya assigning verts to the wrong joints when you start editing weights (in particular, removing them).
So! Some basics I quickly learned working as a rigger in no order ->
* Find or make your own custom smoothing tool.
In my case, my lead had written some really good smoothing tools in python that I've been using at work. Essentially what you want is something that will look at your verts, and smooth out/average out the values based on the edge rings next to it. You can have different versions of this tool as well, for instance if you want a slight smooth, you can look at writing a tool that will average out a vert selection (or obj), based on the verts directly next to it.
If you're not code savvy yet, like me (yet! I will get there, then write about it!), then there are other options. More on this later :)
* Learn anatomy, and if you're working on something stylised - ignore accurate anatomy, and focus on volumes.
The studio I'm in excels at stylized animations, which meant that while yes I know where the anatomically correct positions of bones in a dog leg are, if you place the joints of a dog's backleg accurate to a dogs skeleton in the real world, you are unlikely to get good volume deformation in the leg. Placing bones in the centre of a mass is always a good default if you are unsure.
* Learn how to predict a pivots orientation before binding anything to it.
Especially useful for placing joints for a characters jaw! A good point that was shown to me at work was the rotate tool being a good visual of the arc gained by where your joint is currently placed. (image, wrong placement)
^Can determine whether this pivot point works using the rotate tool. In this case, it does not! The jaw swings too far back compared to what I want ^
In case of the jaw, more times than not for me, the pivot of the jaw actually ends up sitting somewhere in the neck instead of below the ear. (image, correct placement)
* Deformers aren't always necessary.
If you enjoy using wire deformers and all that jazz, by all means use them. They can be useful when an animator plans on horribly breaking things and you don't have fx to save you. But if you don't like using them or can't, don't panic. No one in my rigging team uses deformers, everything is classic skin clusters and blendshapes. Where I work, blendshapes are for the face and are art directed, if you need more muscle definition in the shoulder for example, you can use joints. Deformers can get really tricky when you start to fuse them with other things.
If you need more volume in a specific area like the knees, consider using duel quaternium skinning instead of classic linear-- the weight blended version, so you can paint in your DQ weights over the knees and leave the rest in classic linear. Another way is to think outside the box and use floating joints to hold volume and constrain them accordingly in your rig.
DQ is pretty good with preserving volumes but it doesn't like things scaling, which leads to my next point.
If you can, make sure everyone is on board with sticking to the scale of the project. If your project is about a bunch of ants, then make everything a 10th of the size, don't rely on rigging, layout and anim to constantly scale it down for you (sounds ridiculous, but I've been in little group projects that have done that).
Oh yeah, ever had a model with a really big swollen looking knee joint? Consider using two joints for the knee bend, not one. One that sits at the top of the knee, and another that's just above the bottom of the knee bulge in the model.
- Native Maya - involves more precision and locking weights once they're done
Block out each section. Create a low rez of your model, weight everything to the pelvis. The pelvis joint is like your root, it connects to your master controls, and is the base of your spine.
Now I mentioned locking weights... in my experience, maya likes to throw influences around like it's a party. For instance I'd be working on the shoulder weights, and maya reassigns verts to the foot joint. I've found the best way to avoid this in general is just to lock the joints you aren't working on.
From there, you block out each section of the spine. Solid values, either 0 or 1. Nothing blended, just black and white for now. The Chest bone will also have the head, neck, shoulders, arms, hands and fingers weighted to it. Between your spine bones, choose an edgeloop to represent a 50% way mark between the influences and weight it as so. So if I were to photobash a couple of screenshots...
The initial spine blocking would look like this!
This allows you to see if your joints are correctly placed and gauge whether things will blend well and keep their volume.
After you've blocked out your torso, it's up to you whether you proceed blocking down the legs next or move onto the shoulders and neck, but it's the same principle either way. Once things are blocked out and looking good, smoothing it becomes much easier, and most of your thinking is done.
- Download NG Skin Tools - a layer based approach to skinning (there's a free non commercial version of this plugin)
Circling back to smoothing, ng skin tools has a great smoothing brush as well as a relaxing option. Both are great and work with soft selection too!
This plugin has it's own youtube tutorials that show how to use it, so i won't go into too much depth - I'll just explain what I use it for:
- Assigning weights. This has given me fairly accurate results, and goes off whatever is closest to the joint. I use it all the time for joint chains in characters and props, then just relax and smooth the weights with a couple of clicks and them I'm done.
- Layered skinning - Incredibly useful for when you forget to add joints to your animal's ears! Pop your joints in your rig, make sure they're an influence on the geo, go to ng skin tools, create a new layer and just add in your weights. It's a way you can make additions without being destructive with the weights that you've worked so hard on already. If you don't like your addition, you can always delete it! (con - you can't remove weights added in a layer, however each layer has a mask that you can paint and you can remove values there to hide what you don't want in your layer. Think of it as an opacity map)
- A proper smooth brush.
- The relax option for when I'm feeling really lazy
- Exporting and Importing weights. Occasionally things horribly break and you need to transfer your weights over to new models or rigs. This tool has an export/import weights option that saves your weights as .json files. This one has been surprisingly reliable and provided fairly accurate transfers, but also brings it in as another layer, which is good if you want to mask out areas you don't like in the update.